{"type":"ScholarlyArticle","title":"C3-STISR: Scene Text Image Super-resolution with Triple Clues","authors":[{"name":"Minyi Zhao","slug":"minyi-zhao","given":"Minyi","family":"Zhao","affiliation":"Shanghai Key Lab of Intelligent Information Processing , and School of Computer Science , Fudan University , Shanghai 200438 , China"},{"name":"Miao Wang","slug":"miao-wang","given":"Miao","family":"Wang","affiliation":"ByteDance , China"},{"name":"Fan Bai","slug":"fan-bai","given":"Fan","family":"Bai","affiliation":"Shanghai Key Lab of Intelligent Information Processing , and School of Computer Science , Fudan University , Shanghai 200438 , China"},{"name":"Bingjia Li","slug":"bingjia-li","given":"Bingjia","family":"Li","affiliation":"Shanghai Key Lab of Intelligent Information Processing , and School of Computer Science , Fudan University , Shanghai 200438 , China"},{"name":"Jie Wang","slug":"jie-wang","given":"Jie","family":"Wang","affiliation":"ByteDance , China"},{"name":"Shuigeng Zhou","slug":"shuigeng-zhou","given":"Shuigeng","family":"Zhou","affiliation":"Shanghai Key Lab of Intelligent Information Processing , and School of Computer Science , Fudan University , Shanghai 200438 , China"}],"abstract":"Scene text image super-resolution (STISR) has been regarded as an important pre-processing task for text recognition from low-resolution scene text images. Most recent approaches use the recognizer's feedback as clues to guide super-resolution. However, directly using recognition clue has two problems: 1) Compatibility. It is in the form of probability distribution, has an obvious modal gap with STISR - a pixel-level task; 2) Inaccuracy. it usually contains wrong information, thus will mislead the main task and degrade super-resolution performance. In this paper, we present a novel method C3-STISR that jointly exploits the recognizer's feedback, visual and linguistical information as clues to guide super-resolution. Here, visual clue is from the images of texts predicted by the recognizer, which is informative and more compatible with the STISR task; while linguistical clue is generated by a pre-trained character-level language model, which is able to correct the predicted texts. We design effective extraction and fusion mechanisms for the triple cross-modal clues to generate a comprehensive and unified guidance for super-resolution. Extensive experiments on TextZoom show that C3-STISR outperforms the SOTA methods in fidelity and recognition performance. Code is available in https://github.com/zhaominyiz/C3-STISR.","doi":"10.24963/ijcai.2022/238","url":"https://ijcai.org/proceedings/2022/238","pdf":"https://ijcai.org/proceedings/2022/0238.pdf","markdown":"https://ijcai.org/proceedings/2022/238.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/238","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":238,"pages":{"first":1707,"last":1713},"keywords":["Computer Vision: Applications","Computer Vision: Recognition (object detection, categorization)"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986475","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984618","length_sec":345,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen et al., 2021a] Jingye Chen, Bin Li, and Xiangyang Xue. Scene text telescope: Text-focused scene image super-resolution. In CVPR, pages 12026-12035, 2021.","title":"Scene text telescope: Text-focused scene image super-resolution","authors":"Chen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021b] Jingye Chen, Haiyang Yu, Jianqi Ma, Bin Li, and Xiangyang Xue. Text gestalt: Stroke- aware scene text image super-resolution. arXiv preprint arXiv:2112.08171, 2021.","title":"Text gestalt: Strokeaware scene text image super-resolution","authors":"Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dai et al., 2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. De- formable convolutional networks. In ICCV, pages 764- 773, 2017.","title":"Deformable convolutional networks","authors":"Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2019] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention net- work for single image super-resolution. In CVPR, pages 11065-11074, 2019.","title":"Second-order attention network for single image super-resolution","authors":"Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, Lei Zhang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al., 2015] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. TPAMI, 38(2):295-307, 2015.","title":"Image super-resolution using deep convolutional networks","authors":"Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang","year":"2015","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fang et al., 2021a] Chuantao Fang, Yu Zhu, Lei Liao, and Xiaofeng Ling. Tsrgan: Real-world text image super- resolution based on adversarial learning and triplet atten- tion. Neurocomputing, 455:88-96, 2021.","title":"Tsrgan: Real-world text image superresolution based on adversarial learning and triplet attention","authors":"Fang","year":"2021","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fang et al., 2021b] Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In CVPR, pages 7098-7107, 2021.","title":"Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition","authors":"Fang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In NeurIPS, pages 2017-2025, 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ledig et al., 2017] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super- resolution using a generative adversarial network. In CVPR, pages 4681-4690, 2017.","title":"Photo-realistic single image superresolution using a generative adversarial network","authors":"Ledig","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Luo et al., 2019] Canjie Luo, Lianwen Jin, and Zenghui Sun. Moran: A multi-object rectified attention network for scene text recognition. PR, 90:109-118, 2019.","title":"Moran: A multi-object rectified attention network for scene text recognition","authors":"Luo","year":"2019","venue":"PR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ma et al., 2021] Jianqi Ma, Shi Guo, and Lei Zhang. Text prior guided scene text image super-resolution. arXiv preprint arXiv:2106.15368, 2021.","title":"Text prior guided scene text image super-resolution","authors":"Jianqi Ma, Shi Guo, Lei Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ma et al., 2022] Jianqi Ma, Zhetong Liang, and Lei Zhang. A text attention network for spatial deformation ro- bust scene text image super-resolution. arXiv preprint arXiv:2203.09388, 2022.","title":"A text attention network for spatial deformation robust scene text image super-resolution","authors":"Ma","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Merity et al., 2016] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.","title":"Pointer sentinel mixture models","authors":"Stephen Merity, Caiming Xiong, James Bradbury, Richard Socher","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nakaune et al., 2021] Shimon Nakaune, Satoshi Iizuka, and Kazuhiro Fukui. Skeleton-aware text image super- resolution. In BMVC, pages 1-14, 2021.","title":"Skeleton-aware text image superresolution","authors":"Nakaune","year":"2021","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pandey et al., 2018] Ram Krishna Pandey, K Vignesh, AG Ramakrishnan, et al. Binary document image super resolution for improved readability and ocr performance. arXiv preprint arXiv:1812.02475, 2018.","title":"Binary document image super resolution for improved readability and ocr performance","authors":"Pandey","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shi et al., 2016] Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. TPAMI, 39(11):2298-2304, 2016.","title":"An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition","authors":"Shi","year":"2016","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shi et al., 2018] Baoguang Shi, Mingkun Yang, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai. Aster: An attentional scene text recognizer with flexible rectifi- cation. TPAMI, 41(9):2035-2048, 2018.","title":"Aster: An attentional scene text recognizer with flexible rectification","authors":"Shi","year":"2018","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Singh et al., 2019] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317-8326, 2019.","title":"Towards vqa models that can read","authors":"Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, Marcus Rohrbach","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2019] Wenjia Wang, Enze Xie, Peize Sun, Wenhai Wang, Lixun Tian, Chunhua Shen, and Ping Luo. Textsr: Content-aware text super-resolution guided by recognition. arXiv preprint arXiv:1909.07113, 2019.","title":"Textsr: Content-aware text super-resolution guided by recognition","authors":"Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2020] Wenjia Wang, Enze Xie, Xuebo Liu, Wenhai Wang, Ding Liang, Chunhua Shen, and Xiang Bai. Scene text image super-resolution in the wild. In ECCV, pages 650-666. Springer, 2020.","title":"Scene text image super-resolution in the wild","authors":"Wang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2017] Xiangyu Xu, Deqing Sun, Jinshan Pan, Yujin Zhang, Hanspeter Pfister, and Ming-Hsuan Yang. Learning to super-resolve blurry face and text images. In ICCV, pages 251-260, 2017.","title":"Learning to super-resolve blurry face and text images","authors":"Xiangyu Xu, Deqing Sun, Jinshan Pan, Yujin Zhang, Hanspeter Pfister, Ming-Hsuan Yang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2021] Yi Xu, Minyi Zhao, Jing Liu, Xinjian Zhang, Longwen Gao, Shuigeng Zhou, and Huyang Sun. Boosting the performance of video compression artifact reduction with reference frame proposals and frequency domain information. In CVPRW, pages 213-222, 2021.","title":"Boosting the performance of video compression artifact reduction with reference frame proposals and frequency domain information","authors":"Yi Xu, Minyi Zhao, Jing Liu, Xinjian Zhang, Longwen Gao, Shuigeng Zhou, Huyang Sun","year":"2021","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2018] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention net- works. In ECCV, pages 286-301, 2018.","title":"Image superresolution using very deep residual channel attention networks","authors":"Zhang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2020] Chongsheng Zhang, Weiping Ding, Guowen Peng, Feifei Fu, and Wei Wang. Street view text recognition with deep learning for urban scene understand- ing in intelligent transportation systems. IEEE Transac- tions on Intelligent Transportation Systems, 2020.","title":"Street view text recognition with deep learning for urban scene understanding in intelligent transportation systems","authors":"Chongsheng Zhang, Weiping Ding, Guowen Peng, Feifei Fu, Wei Wang","year":"2020","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhao et al., 2021a] Cairong Zhao, Shuyang Feng, Brian Nlong Zhao, Zhijun Ding, Jun Wu, Fumin Shen, and Heng Tao Shen. Scene text image super-resolution via parallelly contextual attention network. In MM, pages 2908-2917, 2021.","title":"Scene text image super-resolution via parallelly contextual attention network","authors":"Zhao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2021b] Minyi Zhao, Yi Xu, and Shuigeng Zhou. Recursive fusion and deformable spatiotemporal attention for video compression artifact reduction. In MM, pages 5646-5654, 2021.","title":"Recursive fusion and deformable spatiotemporal attention for video compression artifact reduction","authors":"Zhao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}