{"type":"ScholarlyArticle","title":"Reschedule Diffusion-based Bokeh Rendering","authors":[{"name":"Shiyue Yan","slug":"shiyue-yan","given":"Shiyue","family":"Yan","affiliation":"Shenzhen International Graduate School , Tsinghua University"},{"name":"Xiaoshi Qiu","slug":"xiaoshi-qiu","given":"Xiaoshi","family":"Qiu","affiliation":"Shenzhen International Graduate School , Tsinghua University"},{"name":"Qingmin Liao","slug":"qingmin-liao","given":"Qingmin","family":"Liao","affiliation":"Shenzhen International Graduate School , Tsinghua University"},{"name":"Jing-Hao Xue","slug":"jing-hao-xue","given":"Jing-Hao","family":"Xue","affiliation":"Department of Statistical Science , University College London"},{"name":"Shaojun Liu","slug":"shaojun-liu","given":"Shaojun","family":"Liu","affiliation":"College of Health Science and Environmental Engineering , Shenzhen Technology University"}],"abstract":"Bokeh rendering for images shot with small apertures has drawn much attention in practice. Very recently people start to explore diffusion models for bokeh rendering, aiming to leverage the models' surging power of image generation. However, we can clearly observe two big issues with the images rendered by diffusion models: large fluctuation and severe color deviation. To address these issues, we propose in this paper a prior-aware sampling approach, which can adaptively control the noise scale through learned priors, and a prior-aware noise scheduling strategy, which can greatly reduce the number of inference steps without sacrificing performance. Extensive experiments show that our method can effectively alleviate the fluctuation problem of sampling results while ensuring similar color styles to the input image. In addition, our method outperforms state-of-the-art methods, sometimes even with only two steps of sampling. Our code is available at https://github.com/Loeiii/Reschedule-Diffusion-based-Bokeh-Rendering.","doi":"10.24963/ijcai.2024/171","url":"https://ijcai.org/proceedings/2024/171","pdf":"https://ijcai.org/proceedings/2024/0171.pdf","markdown":"https://ijcai.org/proceedings/2024/171.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/171","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":171,"pages":{"first":1543,"last":1551},"keywords":["Computer Vision: CV: Image and video synthesis and generation","Machine Learning: ML: Applications","Humans and AI: HAI: Applications"],"videos":[],"references":[{"position":1,"raw":"[Bae and Durand, 2007] Soonmin Bae and Frédo Durand. Defocus magnification. In Computer graphics forum, vol- ume 26, pages 571-579. Wiley Online Library, 2007.","title":"Soonmin Bae and Frédo Durand. Defocus magnification","authors":"Durand Bae","year":"2007","venue":"Computer graphics forum","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bao et al., 2021] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the op- timal reverse variance in diffusion probabilistic models. In International Conference on Learning Representations, 2021.","title":"Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models","authors":"Fan Bao, Chongxuan Li, Jun Zhu, Bo Zhang","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bao et al., 2022] Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covari- ance with imperfect mean in diffusion probabilistic mod- els. In International Conference on Machine Learning, pages 1555-1584. PMLR, 2022.","title":"Estimating the optimal covariance with imperfect mean in diffusion probabilistic models","authors":"Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, Bo Zhang","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.00713, 2020.","title":"Wavegrad: Estimating gradients for waveform generation","authors":"Chen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Choi et al., 2020] Min-Su Choi, Jun-Hyuk Kim, Jun-Ho Choi, and Jong-Seok Lee. Efficient bokeh effect render- ing using generative adversarial network. In 2020 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia), pages 1-5. IEEE, 2020.","title":"Efficient bokeh effect rendering using generative adversarial network","authors":"Choi","year":"2020","venue":"IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Choi et al., 2022] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472-11481, 2022.","title":"Perception prioritized training of diffusion models","authors":"Choi","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dutta et al., 2021] Saikat Dutta, Sourya Dipta Das, Nis- arg A Shah, and Anil Kumar Tiwari. Stacked deep multi- scale hierarchical network for fast bokeh effect rendering from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2398-2407, 2021.","title":"Stacked deep multiscale hierarchical network for fast bokeh effect rendering from a single image","authors":"Saikat Dutta, Dipta Sourya, Nisarg A Das, Anil Kumar Shah, Tiwari","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Georgiadis et al., 2022] Konstantinos Georgiadis, Albert Saà-Garriga, Mehmet Kerim Yucel, Anastasios Drosou, and Bruno Manganelli. Adaptive mask-based pyramid net- work for realistic bokeh rendering. In European Confer- ence on Computer Vision, pages 429-444. Springer, 2022.","title":"Adaptive mask-based pyramid network for realistic bokeh rendering","authors":"Konstantinos Georgiadis, Albert Saà-Garriga, Mehmet Kerim Yucel, Anastasios Drosou, Bruno Manganelli","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.","title":"Denoising diffusion probabilistic models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ignatov et al., 2020] Andrey Ignatov, Jagruti Patel, and Radu Timofte. Rendering natural camera bokeh effect with deep learning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition Work- shops, pages 418-419, 2020.","title":"Rendering natural camera bokeh effect with deep learning","authors":"Andrey Ignatov, Jagruti Patel, Radu Timofte","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Karras et al., 2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.","title":"Elucidating the design space of diffusion-based generative models","authors":"Tero Karras, Miika Aittala, Timo Aila, Samuli Laine","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2022] Heeseung Kim, Sungwon Kim, and Sun- groh Yoon. Guided-tts: A diffusion model for text-to- speech via classifier guidance. In International Confer- ence on Machine Learning, pages 11119-11133. PMLR, 2022.","title":"Guided-tts: A diffusion model for text-tospeech via classifier guidance","authors":"Kim","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2016] Shaojun Liu, Fei Zhou, and Qingmin Liao. Defocus map estimation from a single image based on two-parameter defocus model. IEEE Transactions on Im- age Processing, 25(12):5943-5956, 2016.","title":"Defocus map estimation from a single image based on two-parameter defocus model","authors":"Shaojun Liu, Fei Zhou, Qingmin Liao","year":"2016","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2022] Yi Liu, Juncheng Zhang, Qingmin Liao, Haoyu Ma, and Shaojun Liu. Feature pyramid boosting network for rendering natural bokeh. In 2022 IEEE In- ternational Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2022.","title":"Feature pyramid boosting network for rendering natural bokeh","authors":"Yi Liu, Juncheng Zhang, Qingmin Liao, Haoyu Ma, Shaojun Liu","year":"2022","venue":"IEEE International Conference on Multimedia and Expo (ICME)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lu et al., 2022] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Process- ing Systems, 35:5775-5787, 2022.","title":"Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps","authors":"Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, Jun Zhu","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luo et al., 2023a] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Image restora- tion with mean-reverting stochastic differential equations. arXiv preprint arXiv:2301.11699, 2023.","title":"Image restoration with mean-reverting stochastic differential equations","authors":"Luo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Luo et al., 2023b] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Refusion: En- abling large-size realistic image restoration with latent- space diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1680-1691, 2023.","title":"Refusion: Enabling large-size realistic image restoration with latentspace diffusion models","authors":"Luo","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nagasubramaniam and Younes, 2022] Hariharan Nagasub- ramaniam and Rabih Younes. Bokeh effect rendering with vision transformers. 2022.","title":"Bokeh effect rendering with vision transformers","authors":"Hariharan Nagasubramaniam, Rabih Younes","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Podell et al., 2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving la- tent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.","title":"Sdxl: Improving latent diffusion models for high-resolution image synthesis","authors":"Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, Robin Rombach","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Purohit et al., 2019] Kuldeep Purohit, Maitreya Suin, Praveen Kandula, and Rajagopalan Ambasamudram. Depth-guided dense dynamic filtering network for bokeh effect rendering. In 2019 IEEE/CVF International Con- ference on Computer Vision Workshop (ICCVW), pages 3417-3426. IEEE, 2019.","title":"Depth-guided dense dynamic filtering network for bokeh effect rendering","authors":"Kuldeep Purohit, Maitreya Suin, Praveen Kandula, Rajagopalan Ambasamudram","year":"2019","venue":"IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Qian et al., 2020] Ming Qian, Congyu Qiao, Jiamin Lin, Zhenyu Guo, Chenghua Li, Cong Leng, and Jian Cheng. Bggan: Bokeh-glass generative adversarial network for rendering realistic bokeh. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Pro- ceedings, Part III 16, pages 229-244. Springer, 2020.","title":"Bggan: Bokeh-glass generative adversarial network for rendering realistic bokeh","authors":"Ming Qian, Congyu Qiao, Jiamin Lin, Zhenyu Guo, Chenghua Li, Cong Leng, Jian Cheng","year":"2020","venue":"Computer Vision-ECCV 2020 Workshops","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.","title":"High-resolution image synthesis with latent diffusion models","authors":"Rombach","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Saharia et al., 2022] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mo- hammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713-4726, 2022.","title":"Image super-resolution via iterative refinement","authors":"Saharia","year":"2022","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Seizinger et al., 2023] Tim Seizinger, Marcos V Conde, Manuel Kolmet, Tom E Bishop, and Radu Timofte. Ef- ficient multi-lens bokeh effect rendering and transforma- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1633- 1642, 2023.","title":"Efficient multi-lens bokeh effect rendering and transformation","authors":"Tim Seizinger, Marcos V Conde, Manuel Kolmet, Tom E Bishop, Radu Timofte","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Song et al., 2020a] Jiaming Song, Chenlin Meng, and Ste- fano Ermon. Denoising diffusion implicit models. In Inter- national Conference on Learning Representations, 2020.","title":"Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models","authors":"Song","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Song et al., 2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Er- mon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.","title":"Score-based generative modeling through stochastic differential equations","authors":"Song","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Voleti et al., 2022] Vikram Voleti, Alexia Jolicoeur- Martineau, and Chris Pal. Mcvd-masked conditional video diffusion for prediction, generation, and interpola- tion. Advances in Neural Information Processing Systems, 35:23371-23385, 2022.","title":"Mcvd-masked conditional video diffusion for prediction, generation, and interpolation","authors":"Vikram Voleti, Alexia Jolicoeur-Martineau, Chris Pal","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wadhwa et al., 2018] Neal Wadhwa, Rahul Garg, David E Jacobs, Bryan E Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T Barron, Yael Pritch, and Marc Levoy. Synthetic depth-of-field with a single- camera mobile phone. ACM Transactions on Graphics (ToG), 37(4):1-13, 2018.","title":"Synthetic depth-of-field with a singlecamera mobile phone","authors":"Neal Wadhwa, Rahul Garg, David E Jacobs, Bryan E Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T Barron, Yael Pritch, Marc Levoy","year":"2018","venue":"ACM Transactions on Graphics (ToG)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xue et al., 2013] Weichen Xue, Xiangze Zhang, Bin Sheng, and Lizhuang Ma. Image-based depth-of-field render- ing with non-local means filtering. In 2013 IEEE Inter- national Conference on Multimedia and Expo Workshops (ICMEW), pages 1-6. IEEE, 2013.","title":"Image-based depth-of-field rendering with non-local means filtering","authors":"Xue","year":"2013","venue":"2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2023] Zhihao Yang, Wenyi Lian, and Siyuan Lai. Bokehornot: Transforming bokeh effect with image transformer and lens metadata embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 1542-1550, 2023.","title":"Bokehornot: Transforming bokeh effect with image transformer and lens metadata embedding","authors":"Zhihao Yang, Wenyi Lian, Siyuan Lai","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ye et al., 2023] Jiasheng Ye, Zaixiang Zheng, Yu Bao, Li- hua Qian, and Mingxuan Wang. Dinoiser: Diffused con- ditional sequence learning by manipulating noises. arXiv preprint arXiv:2302.10025, 2023.","title":"Dinoiser: Diffused conditional sequence learning by manipulating noises","authors":"Ye","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yue et al., 2023] Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super-resolution by residual shifting. arXiv preprint arXiv:2307.12348, 2023.","title":"Resshift: Efficient diffusion model for image super-resolution by residual shifting","authors":"Yue","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang and Chen, 2022] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902, 2022.","title":"Fast sampling of diffusion models with exponential integrator","authors":"Chen ; Zhang, Qinsheng Zhang, Yongxin Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image dif- fusion models. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 3836-3847, 2023.","title":"Adding conditional control to text-to-image diffusion models","authors":"Zhang","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhuo and Sim, 2011] Shaojie Zhuo and Terence Sim. Defo- cus map estimation from a single image. Pattern Recogni- tion, 44(9):1852-1858, 2011.","title":"Shaojie Zhuo and Terence Sim. Defocus map estimation from a single image","authors":"Sim Zhuo","year":"2011","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}