{"type":"ScholarlyArticle","title":"A Decoder-free Transformer-like Architecture for High-efficiency Single Image Deraining","authors":[{"name":"Xiao Wu","slug":"xiao-wu","given":"Xiao","family":"Wu","affiliation":"University of Electronic Science and Technology of China , Chengdu , 611731"},{"name":"Ting-Zhu Huang","slug":"ting-zhu-huang","given":"Ting-Zhu","family":"Huang","affiliation":"University of Electronic Science and Technology of China , Chengdu , 611731"},{"name":"Liang-Jian Deng","slug":"liang-jian-deng","given":"Liang-Jian","family":"Deng","affiliation":"University of Electronic Science and Technology of China , Chengdu , 611731"},{"name":"Tian-Jing Zhang","slug":"tian-jing-zhang","given":"Tian-Jing","family":"Zhang","affiliation":"University of Electronic Science and Technology of China , Chengdu , 611731"}],"abstract":"Despite the success of vision Transformers for the image deraining task, they are limited by computation-heavy and slow runtime. In this work, we investigate Transformer decoder is not necessary and has huge computational costs. Therefore, we revisit the standard vision Transformer as well as its successful variants and propose a novel Decoder-Free Transformer-Like (DFTL) architecture for fast and accurate single image deraining. Specifically, we adopt a cheap linear projection to represent visual information with lower\r\ncomputational costs than previous linear projections. Then we replace standard Transformer decoder block with designed Progressive Patch Merging (PPM), which attains comparable performance and efficiency. DFTL could significantly alleviate the computation and GPU memory requirements through proposed modules. Extensive experiments\r\ndemonstrate the superiority of DFTL compared with competitive Transformer architectures, e.g., ViT, DETR, IPT, Uformer, and Restormer. The code is available at https://github.com/XiaoXiao-Woo/derain.","doi":"10.24963/ijcai.2022/205","url":"https://ijcai.org/proceedings/2022/205","pdf":"https://ijcai.org/proceedings/2022/0205.pdf","markdown":"https://ijcai.org/proceedings/2022/205.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/205","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":205,"pages":{"first":1474,"last":1480},"keywords":["Machine Learning: Other","Machine Learning: Applications","Computer Vision: Machine Learning for Vision","Machine Learning: Convolutional Networks"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987018","length_sec":81,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985161","length_sec":619,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Carion et al., 2020] N Carion, F Massa, G Synnaeve, N Usunier, A Kirillov, and S Zagoruyko. End-to-end ob- ject detection with transformers. In ECCV, pages 213- 229, 2020.","title":"End-to-end object detection with transformers","authors":"Carion, G Massa, N Synnaeve, Usunier, Kirillov, Zagoruyko","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, pages 12299-12310, 2021.","title":"Pre-trained image processing transformer","authors":"Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, Wen Gao","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2018] Liang-Jian Deng, Ting-Zhu Huang, Xi- Le Zhao, and Tai-Xiang Jiang. A directional global sparse model for single image rain removal. Applied Mathemati- cal Modelling, 59:662-679, 2018.","title":"A directional global sparse model for single image rain removal","authors":"Liang-Jian Deng, Ting-Zhu Huang, Xi-Le Zhao, Tai-Xiang Jiang","year":"2018","venue":"Applied Mathematical Modelling","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2017a] Xue-Yang Fu, Jia-Bin Huang, Zeng De- Lu, Yue Huang, Xing-Hao Ding, and John Paisley. Re- moving rain from single images via a deep detail network. In CVPR, pages 1715-1723, 2017.","title":"Removing rain from single images via a deep detail network","authors":"Fu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu et al., 2017b] Xue-Yang Fu, Jia-Bin Huang, Xing-Hao Ding, Ying-Hao Liao, and John Paisley. Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing, 26(6):2944-2956, 2017.","title":"Clearing the skies: A deep network architecture for single-image rain removal","authors":"Fu","year":"2017","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fu et al., 2021] Xue-Yang Fu, Qi Qi, Zheng-Jun Zha, Yu- Rui Zhu, and Xing-Hao Ding. Rain streak removal via dual graph convolutional network. In AAAI, pages 1352- 1360, 2021.","title":"Rain streak removal via dual graph convolutional network","authors":"Xue-Yang Fu, Qi Qi, Zheng-Jun Zha, Yu-Rui Zhu, Xing-Hao Ding","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gu et al., 2017] Shu-Hang Gu, De-Yu Meng, Wang-Meng Zuo, and Lei Zhang. Joint convolutional analysis and syn- thesis sparse representation for single image layer separa- tion. In ICCV, pages 1717-1725, 2017.","title":"Joint convolutional analysis and synthesis sparse representation for single image layer separation","authors":"Shu-Hang Gu, De-Yu Meng, Wang-Meng Zuo, Lei Zhang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2017] Tai-Xiang Jiang, Ting-Zhu Huang, Xi-Le Zhao, Liang-Jian Deng, and Yao Wang. A novel tensor- based video rain streaks removal approach via utilizing discriminatively intrinsic priors. In CVPR, pages 2818- 2827, 2017.","title":"A novel tensorbased video rain streaks removal approach via utilizing discriminatively intrinsic priors","authors":"Tai-Xiang Jiang, Ting-Zhu Huang, Xi-Le Zhao, Liang-Jian Deng, Yao Wang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2019] Tai-Xiang Jiang, Ting-Zhu Huang, Xi-Le Zhao, Liang-Jian Deng, and Yao Wang. Fastderain: A novel video rain streak removal method using directional gradient priors. IEEE Transactions on Image Processing, 28(4):2089-2102, 2019.","title":"Fastderain: A novel video rain streak removal method using directional gradient priors","authors":"Tai-Xiang Jiang, Ting-Zhu Huang, Xi-Le Zhao, Liang-Jian Deng, Yao Wang","year":"2019","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2016] Yu Li, Robby T Tan, Guo Xiao-Jie, Lu Jiang-Bo, and Michael S Brown. Rain streak removal using layer priors. In CVPR, pages 2736-2744, 2016.","title":"Rain streak removal using layer priors","authors":"Yu Li, Robby T Tan, Guo Xiao-Jie, Lu Jiang-Bo, Michael S Brown","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2018] Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. Recurrent squeeze-and-excitation context aggregation net for single image deraining. In ECCV, pages 262-277, 2018.","title":"Recurrent squeeze-and-excitation context aggregation net for single image deraining","authors":"Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, Hongbin Zha","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liang et al., 2021] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCV, pages 1833-1844, 2021.","title":"Swinir: Image restoration using swin transformer","authors":"Liang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luo et al., 2015] Yu Luo, Yong Xu, and Hui Ji. Removing rain from a single image via discriminative sparse coding. In ICCV, pages 3397-3405, 2015.","title":"Removing rain from a single image via discriminative sparse coding","authors":"Yu Luo, Yong Xu, Hui Ji","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ren et al., 2019] Dong-Wei Ren, Wang-Meng Zuo, Qing- Hua Hu, Peng-Fei Zhu, and De-Yu Meng. Progressive im- age deraining networks: A better and simpler baseline. In CVPR, pages 3932-3941, 2019.","title":"Progressive image deraining networks: A better and simpler baseline","authors":"Dong-Wei Ren, Wang-Meng Zuo, Qing-Hua Hu, Peng-Fei Zhu, De-Yu Meng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeuIPS, page 6000-6010, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeuIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2004] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"A C Zhou Wang, H R Bovik, E P Sheikh, Simoncelli","year":"2004","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2020] Hong Wang, Qi Xie, Qian Zhao, and De- Yu Meng. A model-driven deep neural network for single image rain removal. In CVPR, pages 3100-3109, 2020.","title":"A model-driven deep neural network for single image rain removal","authors":"Wang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2021a] Wen-Hai Wang, En-Ze Xie, Xiang Li, Deng-Ping Fan, Kai-Tao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A ver- satile backbone for dense prediction without convolutions. In ICCV, 2021.","title":"Pyramid vision transformer: A versatile backbone for dense prediction without convolutions","authors":"Wang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2021b] Zhen-Dong Wang, Xiao-Dong Cun, Jian-Min Bao, and Jian-Zhuang Liu. Uformer: A general u-shaped transformer for image restoration. arXiv preprint arXiv:2106.03106, 2021.","title":"Uformer: A general u-shaped transformer for image restoration","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2020] Wenhan Yang, Shiqi Wang, Dejia Xu, Xi- aodong Wang, and Jiaying Liu. Towards scale-free rain streak removal via self-supervised fractal band learning. In AAAI, volume 34, pages 12629-12636, 2020.","title":"Towards scale-free rain streak removal via self-supervised fractal band learning","authors":"Yang","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zamir et al., 2021] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. arXiv preprint arXiv:2111.09881, 2021.","title":"Restormer: Efficient transformer for high-resolution image restoration","authors":"Aditya Syed Waqas Zamir, Salman Arora, Munawar Khan, Fahad Hayat, Ming-Hsuan Shahbaz Khan, Yang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2017] He Zhang, Sindagi Vishwanath, and Vishal M Patel. Image de-raining using a conditional gen- erative adversarial network. IEEE Transactions on Cir- cuits and Systems for Video Technology, 2017.","title":"Image de-raining using a conditional generative adversarial network","authors":"Zhang","year":"2017","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhu et al., 2021] Xi-Zhou Zhu, Wei-Jie Su, Le-Wei Lu, Bin Li, Xiao-Gang Wang, and Ji-Feng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.","title":"Deformable detr: Deformable transformers for end-to-end object detection","authors":"Xi-Zhou Zhu, Wei-Jie Su, Le-Wei Lu, Bin Li, Xiao-Gang Wang, Ji-Feng Dai","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}