{"type":"ScholarlyArticle","title":"Pyramid Diffusion Models for Low-light Image Enhancement","authors":[{"name":"Dewei Zhou","slug":"dewei-zhou","given":"Dewei","family":"Zhou","affiliation":"ReLER , CCAI , Zhejiang University"},{"name":"Zongxin Yang","slug":"zongxin-yang","given":"Zongxin","family":"Yang","affiliation":"ReLER , CCAI , Zhejiang University"},{"name":"Yi Yang","slug":"yi-yang","given":"Yi","family":"Yang","affiliation":"ReLER , CCAI , Zhejiang University"}],"abstract":"Recovering noise-covered details from low-light images is challenging, and the results given by previous methods leave room for improvement. Recent diffusion models show realistic and detailed image generation through a sequence of denoising refinements and motivate us to introduce them to low-light image enhancement for recovering realistic details. However, we found two problems when doing this, i.e., 1) diffusion models keep constant resolution in one reverse process, which limits the speed; 2) diffusion models sometimes result in global degradation (e.g., RGB shift). To address the above problems, this paper proposes a Pyramid Diffusion model (PyDiff) for low-light image enhancement. PyDiff uses a novel pyramid diffusion method to perform sampling in a pyramid resolution style (i.e., progressively increasing resolution in one reverse process). Pyramid diffusion makes PyDiff much faster than vanilla diffusion models and introduces no performance degradation. Furthermore, PyDiff uses a global corrector to alleviate the global degradation that may occur in the reverse process, significantly improving the performance and making the training of diffusion models easier with little additional computational consumption. Extensive experiments on popular benchmarks show that PyDiff achieves superior performance and efficiency. Moreover, PyDiff can generalize well to unseen noise and illumination distributions. Code and supplementary materials are available at https://github.com/limuloo/PyDIff.git.","doi":"10.24963/ijcai.2023/199","url":"https://ijcai.org/proceedings/2023/199","pdf":"https://ijcai.org/proceedings/2023/0199.pdf","markdown":"https://ijcai.org/proceedings/2023/199.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/199","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":199,"pages":{"first":1795,"last":1803},"keywords":["Computer Vision: CV: Neural generative models, auto encoders, GANs","Computer Vision: CV: Computational photography"],"videos":[],"references":[{"position":1,"raw":"[Bao et al., 2022] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the op- timal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.06503, 2022.","title":"Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models","authors":"Fan Bao, Chongxuan Li, Jun Zhu, Bo Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In Proceed- ings of the IEEE conference on computer vision and pat- tern recognition, pages 3291-3300, 2018.","title":"Learning to see in the dark","authors":"Chen Chen, Qifeng Chen, Jia Xu, Vladlen Koltun","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Choi et al., 2021] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. In 2021 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 14347-14356. IEEE, 2021.","title":"Ilvr: Conditioning method for denoising diffusion probabilistic models","authors":"Choi","year":"2021","venue":"IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cui et al., 2022] Ziteng Cui, Kunchang Li, Lin Gu, Sheng- han Su, Peng Gao, Zhengkai Jiang, Yu Qiao, and Tatsuya Harada. Illumination adaptive transformer. arXiv preprint arXiv:2205.14871, 2022.","title":"Illumination adaptive transformer","authors":"Ziteng Cui, Kunchang Li, Lin Gu, Shenghan Su, Peng Gao, Zhengkai Jiang, Yu Qiao, Tatsuya Harada","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dhariwal and Nichol, 2021] Prafulla Dhariwal and Alexan- der Nichol. Diffusion models beat gans on image synthe- sis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.","title":"Diffusion models beat gans on image synthesis","authors":"Nichol Dhariwal","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dinh et al., 2016] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016.","title":"Density estimation using real nvp","authors":"Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fan et al., 2022a] Chi-Mao Fan, Tsung-Jung Liu, and Kuan-Hsien Liu. Half wavelet attention on m-net+ for low-light image enhancement. arXiv preprint arXiv:2203.01296, 2022.","title":"Half wavelet attention on m-net+ for low-light image enhancement","authors":"Fan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fan et al., 2022b] Wan-Cyuan Fan, Yen-Chun Chen, Dong- Dong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang. Frido: Feature pyramid diffusion for complex scene image synthesis. arXiv preprint arXiv:2208.13753, 2022.","title":"Frido: Feature pyramid diffusion for complex scene image synthesis","authors":"Fan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo and Hu, 2022] Xiaojie Guo and Qiming Hu. Low-light image enhancement via breaking down the darkness. In- ternational Journal of Computer Vision, pages 1-19, 2022.","title":"Low-light image enhancement via breaking down the darkness","authors":"Xiaojie Guo, Qiming Hu","year":"2022","venue":"ternational Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2020] Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong. Zero-reference deep curve estimation for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1780-1789, 2020.","title":"Zero-reference deep curve estimation for low-light image enhancement","authors":"Guo","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2020] Jingwen He, Yihao Liu, Yu Qiao, and Chao Dong. Conditional sequential modulation for efficient global image retouching. In European Conference on Computer Vision, pages 679-695. Springer, 2020.","title":"Conditional sequential modulation for efficient global image retouching","authors":"Jingwen He, Yihao Liu, Yu Qiao, Chao Dong","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.","title":"Denoising diffusion probabilistic models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ho et al., 2022] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Sal- imans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23:47-1, 2022.","title":"Cascaded diffusion models for high fidelity image generation","authors":"Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, Tim Salimans","year":"2022","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jiang et al., 2021] Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. Enlightengan: Deep light en- hancement without paired supervision. IEEE Transactions on Image Processing, 30:2340-2349, 2021.","title":"Enlightengan: Deep light enhancement without paired supervision","authors":"Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, Zhangyang Wang","year":"2021","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jin et al., 2022] Yeying Jin, Wenhan Yang, and Robby T Tan. Unsupervised night image enhancement: When layer decomposition meets light-effects suppression. In Eu- ropean Conference on Computer Vision, pages 404-421. Springer, 2022.","title":"Unsupervised night image enhancement: When layer decomposition meets light-effects suppression","authors":"Jin","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim et al., 2021] Hanul Kim, Su-Min Choi, Chang-Su Kim, and Yeong Jun Koh. Representative color trans- form for image enhancement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4459-4468, 2021.","title":"Representative color transform for image enhancement","authors":"Hanul Kim, Su-Min Choi, Chang-Su Kim, Yeong Jun, Koh","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kingma and Dhariwal, 2018] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 con- volutions. Advances in neural information processing sys- tems, 31, 2018.","title":"Glow: Generative flow with invertible 1x1 convolutions","authors":"Dhariwal ; Kingma, P Durk, Prafulla Kingma, Dhariwal","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Edwin H Land. The retinex theory of color vi- sion. Scientific american, 237(6):108-129, 1977.","title":"The retinex theory of color vision","authors":"H Edwin, Land","year":"1977","venue":"Scientific american","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lore et al., 2017] Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar. Llnet: A deep autoencoder approach to natural low-light image enhancement. Pattern Recogni- tion, 61:650-662, 2017.","title":"Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar. Llnet: A deep autoencoder approach to natural low-light image enhancement","authors":"Lore","year":"2017","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lu et al., 2022] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927, 2022.","title":"Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps","authors":"Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, Jun Zhu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nichol and Dhariwal, 2021] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion proba- bilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.","title":"Improved denoising diffusion probabilistic models","authors":"Nichol, Dhariwal, Alexander Quinn, Nichol, Prafulla Dhariwal","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.","title":"High-resolution image synthesis with latent diffusion models","authors":"Rombach","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Saharia et al., 2022a] Chitwan Saharia, William Chan, Hui- wen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.","title":"Palette: Image-to-image diffusion models","authors":"Saharia","year":"2022","venue":"ACM SIGGRAPH 2022 Conference Proceedings","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Saharia et al., 2022b] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mo- hammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.","title":"Image super-resolution via iterative refinement","authors":"Saharia","year":"2022","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Song et al., 2020a] Jiaming Song, Chenlin Meng, and Ste- fano Ermon. Denoising diffusion implicit models. In Proc. of ICLR, 2020.","title":"Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models","authors":"Song","year":"2020","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Song et al., 2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Confer- ence on Learning Representations, 2020.","title":"Score-based generative modeling through stochastic differential equations","authors":"Song","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans- actions on Image Processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2022] Yufei Wang, Renjie Wan, Wenhan Yang, Haoliang Li, Lap-Pui Chau, and Alex Kot. Low-light im- age enhancement with normalizing flow. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2604-2612, 2022.","title":"Low-light image enhancement with normalizing flow","authors":"Yufei Wang, Renjie Wan, Wenhan Yang, Haoliang Li, Lap-Pui Chau, Alex Kot","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wei et al., 2018] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560, 2018.","title":"Deep retinex decomposition for low-light enhancement","authors":"Chen Wei, Wenjing Wang, Wenhan Yang, Jiaying Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Whang et al., 2022] Jay Whang, Mauricio Delbracio, Hos- sein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refine- ment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16293- 16303, 2022.","title":"Deblurring via stochastic refinement","authors":"Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, Peyman Milanfar","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2021] Wenhan Yang, Wenjing Wang, Haofeng Huang, Shiqi Wang, and Jiaying Liu. Sparse gradient reg- ularized deep retinex network for robust low-light image enhancement. IEEE Transactions on Image Processing, 30:2072-2086, 2021.","title":"Sparse gradient regularized deep retinex network for robust low-light image enhancement","authors":"Yang","year":"2021","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018.","title":"The unreasonable effectiveness of deep features as a perceptual metric","authors":"Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, Oliver Wang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2019] Yonghua Zhang, Jiawan Zhang, and Xi- aojie Guo. Kindling the darkness: A practical low-light image enhancer. In Proceedings of the 27th ACM interna- tional conference on multimedia, pages 1632-1640, 2019.","title":"Kindling the darkness: A practical low-light image enhancer","authors":"Yonghua Zhang, Jiawan Zhang, Xiaojie Guo","year":"2019","venue":"Proceedings of the 27th ACM international conference on multimedia","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2021] Yonghua Zhang, Xiaojie Guo, Jiayi Ma, Wei Liu, and Jiawan Zhang. Beyond brightening low- light images. International Journal of Computer Vision, 129(4):1013-1037, 2021.","title":"Beyond brightening lowlight images","authors":"Yonghua Zhang, Xiaojie Guo, Jiayi Ma, Wei Liu, Jiawan Zhang","year":"2021","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}