{"type":"ScholarlyArticle","title":"Dual Prompt Learning for Continual Rain Removal from Single Images","authors":[{"name":"Minghao Liu","slug":"minghao-liu","given":"Minghao","family":"Liu","affiliation":"Wangxuan Institute of Computer Technology , Peking University; School of EECS , Peking University; School of CS , Peking University"},{"name":"Wenhan Yang","slug":"wenhan-yang","given":"Wenhan","family":"Yang","affiliation":"Peng Cheng Laboratory"},{"name":"Yuzhang Hu","slug":"yuzhang-hu","given":"Yuzhang","family":"Hu","affiliation":"Wangxuan Institute of Computer Technology , Peking University"},{"name":"Jiaying Liu","slug":"jiaying-liu","given":"Jiaying","family":"Liu","affiliation":"Wangxuan Institute of Computer Technology , Peking University"}],"abstract":"Recent efforts have achieved remarkable progress on single image deraining on the stationary distributed data. However, catastrophic forgetting raises practical concerns when applying these methods to real applications, where the data distributions change constantly. In this paper, we investigate the continual learning issue for rain removal and develop a novel efficient continual learned deraining transformer. Different from the typical replay or regularization-based methods that increase overall training time or parameter space, our method relies on compact prompts which are learnable parameters, to maintain both task-invariant and task-specific knowledge. Our prompts are applied at both image and feature levels to leverage effectively transferred knowledge of images and features among different tasks. We conduct comprehensive experiments under widely-used rain removal datasets, where our proposed dual prompt learning consistently outperforms prior state-of-the-art methods. Moreover, we observe that, even though our method is designed for continual learning, it still achieves superior results on the stationary distributed data, which further demonstrates the effectiveness of our method. Our website is available at: http://liuminghao.com.cn/DPL/.","doi":"10.24963/ijcai.2023/849","url":"https://ijcai.org/proceedings/2023/849","pdf":"https://ijcai.org/proceedings/2023/0849.pdf","markdown":"https://ijcai.org/proceedings/2023/849.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/849","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Late Papers","track":"Main Track","number":849,"pages":{"first":7215,"last":7223},"keywords":["Computer Vision: CV: Computational photography"],"videos":[],"references":[{"position":1,"raw":"[Aljundi et al., 2018] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory Aware Synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.","title":"Memory Aware Synapses: Learning what (not) to forget","authors":"Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, Tinne Tuytelaars","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for se- mantic image segmentation. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), 2018.","title":"Encoder-decoder with atrous separable convolution for semantic image segmentation","authors":"Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, Hartwig Adam","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[De Lange et al., 2021] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gre- gory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelli- gence (TPAMI), 2021.","title":"A continual learning survey: Defying forgetting in classification tasks","authors":"Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, Tinne Tuytelaars","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2017] Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. Removing rain from single images via a deep detail network. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.","title":"Removing rain from single images via a deep detail network","authors":"Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, John Paisley","year":"2017","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hu et al., 2021] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. arXiv:2106.09685, 2021.","title":"LoRA: Low-rank adaptation of large language models","authors":"Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kirkpatrick et al., 2017] James Kirkpatrick, Razvan Pas- canu, Neil Rabinowitz, Joel Veness, Guillaume Des- jardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceed- ings of the National Academy of Sciences (PNAS), 2017.","title":"Overcoming catastrophic forgetting in neural networks","authors":"James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska","year":"2017","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lester et al., 2021] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv:2104.08691, 2021.","title":"The power of scale for parameter-efficient prompt tuning","authors":"Brian Lester, Rami Al-Rfou, Noah Constant","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li and Hoiem, 2017] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pat- tern Analysis and Machine Intelligence (TPAMI), 2017.","title":"Learning without forgetting","authors":"Hoiem ; Li, Zhizhong Li, Derek Hoiem","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li and Liang, 2021] Xiang Lisa Li and Percy Liang. Prefix- tuning: Optimizing continuous prompts for generation. arXiv:2101.00190, 2021.","title":"Prefixtuning: Optimizing continuous prompts for generation","authors":"Lisa Xiang, Percy Li, Liang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2018a] Guanbin Li, Xiang He, Wei Zhang, Huiyou Chang, Le Dong, and Liang Lin. Non-locally enhanced encoder-decoder network for single image de-raining. In Proceedings of the ACM International Conference on Mul- timedia (ACM MM), 2018.","title":"Non-locally enhanced encoder-decoder network for single image de-raining","authors":"Li","year":"2018","venue":"Proceedings of the ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2018b] Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. Recurrent squeeze-and-excitation context aggregation net for single image deraining. In Pro- ceedings of the European conference on computer vision (ECCV), 2018.","title":"Recurrent squeeze-and-excitation context aggregation net for single image deraining","authors":"Li","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2021] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neu- big. Pre-train, prompt, and predict: A systematic sur- vey of prompting methods in natural language processing. arXiv:2107.13586, 2021.","title":"Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing","authors":"Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, Graham Neubig","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mai et al., 2022] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing, 2022.","title":"Online continual learning in image classification: An empirical survey","authors":"Mai","year":"2022","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mallya et al., 2018] Arun Mallya, Dillon Davis, and Svet- lana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.","title":"Piggyback: Adapting a single network to multiple tasks by learning to mask weights","authors":"Arun Mallya, Dillon Davis, Svetlana Lazebnik","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pascanu and Bengio, 2013] Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv:1301.3584, 2013.","title":"Revisiting natural gradient for deep networks","authors":"Razvan Pascanu, Yoshua Bengio","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pfeiffer et al., 2020] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv:2005.00247, 2020.","title":"Adapterfusion: Non-destructive task composition for transfer learning","authors":"Pfeiffer","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2020","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ren et al., 2019] Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image de- raining networks: A better and simpler baseline. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.","title":"Progressive image deraining networks: A better and simpler baseline","authors":"Ren","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Valanarasu et al., 2022] Jeya Maria Jose Valanarasu, Rajeev Yasarla, and Vishal M. Patel. Transweather: Transformer- based restoration of images degraded by adverse weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.","title":"Transweather: Transformerbased restoration of images degraded by adverse weather conditions","authors":"Jeya Maria, Jose Valanarasu, Rajeev Yasarla, Vishal M Patel","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2004] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing (TIP), 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"A C Zhou Wang, H R Bovik, E P Sheikh, Simoncelli","year":"2004","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2020] Lijun Wang, Jianming Zhang, Oliver Wang, Zhe Lin, and Huchuan Lu. SDC-depth: Seman- tic divide-and-conquer network for monocular depth esti- mation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.","title":"SDC-depth: Semantic divide-and-conquer network for monocular depth estimation","authors":"Lijun Wang, Jianming Zhang, Oliver Wang, Zhe Lin, Huchuan Lu","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2022] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.","title":"Learning to prompt for continual learning","authors":"Wang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu and Zhu, 2018] Ju Xu and Zhanxing Zhu. Reinforced continual learning. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), 2018.","title":"Reinforced continual learning","authors":"Zhu ; ] Ju Xu, Zhanxing Xu, Zhu","year":"2018","venue":"Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2017] Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from a single image. In Proceedings of IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), 2017.","title":"Deep joint rain detection and removal from a single image","authors":"Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, Shuicheng Yan","year":"2017","venue":"Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2019] Wenhan Yang, Robby T. Tan, Jiashi Feng, Zongming Guo, Shuicheng Yan, and Jiaying Liu. Joint rain detection and removal from a single image with contextualized deep networks. IEEE Transactions on Pat- tern Analysis and Machine Intelligence (TPAMI), 2019.","title":"Joint rain detection and removal from a single image with contextualized deep networks","authors":"Wenhan Yang, Robby T Tan, Jiashi Feng, Zongming Guo, Shuicheng Yan, Jiaying Liu","year":"2019","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zamir et al., 2021] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progres- sive image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.","title":"Multi-stage progressive image restoration","authors":"Aditya Syed Waqas Zamir, Salman Arora, Munawar Khan, Fahad Hayat, Ming-Hsuan Shahbaz Khan, Ling Yang, Shao","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zenke et al., 2017] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic in- telligence. In Proceedings of International Conference on Machine Learning (ICML), 2017.","title":"Continual learning through synaptic intelligence","authors":"Friedemann Zenke, Ben Poole, Surya Ganguli","year":"2017","venue":"Proceedings of International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang and Patel, 2018] He Zhang and Vishal M. Patel. Density-aware single image de-raining using a multi- stream dense network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.","title":"Density-aware single image de-raining using a multistream dense network","authors":"Patel ; ] He Zhang, Vishal M Zhang, Patel","year":"2018","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2019] He Zhang, Vishwanath Sindagi, and Vishal M. Patel. Image de-raining using a conditional gen- erative adversarial network. IEEE Transactions on Cir- cuits and Systems for Video Technology (TCSVT), 2019.","title":"Image de-raining using a conditional generative adversarial network","authors":"Zhang","year":"2019","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2020] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C.-C. Jay Kuo. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2020.","title":"Class-incremental learning via deep model consolidation","authors":"Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, C.-C. Jay Kuo","year":"2020","venue":"Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhao et al., 2021] Hanbin Zhao, Hui Wang, Yongjian Fu, Fei Wu, and Xi Li. Memory efficient class-incremental learning for image classification. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2021.","title":"Memory efficient class-incremental learning for image classification","authors":"Hanbin Zhao, Hui Wang, Yongjian Fu, Fei Wu, Xi Li","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou et al., 2021] Man Zhou, Jie Xiao, Yifan Chang, Xueyang Fu, Aiping Liu, Jinshan Pan, and Zheng-Jun Zha. Image de-raining via continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), 2021.","title":"Image de-raining via continual learning","authors":"Zhou","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}