{"type":"ScholarlyArticle","title":"SVTR: Scene Text Recognition with a Single Visual Model","authors":[{"name":"Yongkun Du","slug":"yongkun-du","given":"Yongkun","family":"Du","affiliation":"School of Computer and Information Technology and Beijing Key Lab of Traffic Data Analysis and Mining , Beijing Jiaotong University , China"},{"name":"Zhineng Chen","slug":"zhineng-chen","given":"Zhineng","family":"Chen","affiliation":"Shanghai Collaborative Innovation Center of Intelligent Visual Computing , School of Computer Science , Fudan University , China"},{"name":"Caiyan Jia","slug":"caiyan-jia","given":"Caiyan","family":"Jia","affiliation":"School of Computer and Information Technology and Beijing Key Lab of Traffic Data Analysis and Mining , Beijing Jiaotong University , China"},{"name":"Xiaoting Yin","slug":"xiaoting-yin","given":"Xiaoting","family":"Yin","affiliation":"Baidu Inc. , China"},{"name":"Tianlun Zheng","slug":"tianlun-zheng","given":"Tianlun","family":"Zheng","affiliation":"Shanghai Collaborative Innovation Center of Intelligent Visual Computing , School of Computer Science , Fudan University , China"},{"name":"Chenxia Li","slug":"chenxia-li","given":"Chenxia","family":"Li","affiliation":"Baidu Inc. , China"},{"name":"Yuning Du","slug":"yuning-du","given":"Yuning","family":"Du","affiliation":"Baidu Inc. , China"},{"name":"Yu-Gang Jiang","slug":"yu-gang-jiang","given":"Yu-Gang","family":"Jiang","affiliation":"Shanghai Collaborative Innovation Center of Intelligent Visual Computing , School of Computer Science , Fudan University , China"}],"abstract":"Dominant scene text recognition models commonly contain two building blocks, a visual model for feature extraction and a sequence model for text transcription. This hybrid architecture, although accurate, is complex and less efficient. In this study, we propose a Single Visual model for Scene Text recognition within the patch-wise image tokenization framework, which dispenses with the sequential modeling entirely. The method, termed SVTR, firstly decomposes an image text into small patches named character components. Afterward, hierarchical stages are recurrently carried out by component-level mixing, merging and/or combining. Global and local mixing blocks are devised to perceive the inter-character and intra-character patterns, leading to a multi-grained character component perception. Thus, characters are recognized by a simple linear prediction. Experimental results on both English and Chinese scene text recognition tasks demonstrate the effectiveness of SVTR. SVTR-L (Large) achieves highly competitive accuracy in English and outperforms existing methods by a large margin in Chinese, while running faster. In addition, SVTR-T (Tiny) is an effective and much smaller model, which shows appealing speed at inference. The code is publicly available at https://github.com/PaddlePaddle/PaddleOCR.","doi":"10.24963/ijcai.2022/124","url":"https://ijcai.org/proceedings/2022/124","pdf":"https://ijcai.org/proceedings/2022/0124.pdf","markdown":"https://ijcai.org/proceedings/2022/124.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/124","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":124,"pages":{"first":884,"last":890},"keywords":["Computer Vision: Scene analysis and understanding","Computer Vision: Recognition (object detection, categorization)"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987117","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985260","length_sec":703,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Anhar et al., 2014] R. Anhar, S. Palaiahnakote, C. S. Chan, and C. L. Tan. A robust arbitrary text detection system for natural scene images. In Expert Systems with Applications, page 8027-8048, 2014.","title":"A robust arbitrary text detection system for natural scene images","authors":"R Anhar, S Palaiahnakote, C S Chan, C L Tan","year":"2014","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Atienza, 2021] R. Atienza. Vision transformer for fast and efficient scene text recognition. arXiv:2105.08582, 2021.","title":"Vision transformer for fast and efficient scene text recognition","authors":"R Atienza","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baek et al., 2019] J. Baek, G. Kim, J. Lee, S. Park, D. Han, S. Yun, S. J. Oh, and H. Lee. What is wrong with scene text recognition model comparisons? dataset and model analysis. In ICCV, pages 4714-4722, 2019.","title":"What is wrong with scene text recognition model comparisons? dataset and model analysis","authors":"J Baek, G Kim, J Lee, S Park, D Han, S Yun, S J Oh, H Lee","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Borisyuk et al., 2018] F. Borisyuk, A. Gordo, and V. Sivakumar. Rosetta: Large scale system for text detection and recognition in images. In ACM SIGKDD, page 71-79, 2018.","title":"Rosetta: Large scale system for text detection and recognition in images","authors":"F Borisyuk, A Gordo, V Sivakumar","year":"2018","venue":"ACM SIGKDD","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2021] J. Chen, H. Yu, J. Ma, M. Guan, X. Xu, X. Wang, S. Qu, B. Li, and X. Xue. Benchmarking chi- nese text recognition: Datasets, baselines, and an empiri- cal study. arXiv:2112.15093, 2021.","title":"Benchmarking chinese text recognition: Datasets, baselines, and an empirical study","authors":"J Chen, H Yu, J Ma, M Guan, X Xu, X Wang, S Qu, B Li, X Xue","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dosovitskiy et al., 2021] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"A Dosovitskiy, L Beyer, A Kolesnikov, D Weissenborn, X Zhai, T Unterthiner, M Dehghani, M Minderer, G Heigold, S Gelly, J Uszkoreit, N Houlsby","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fang et al., 2021] S. Fang, H. Xie, Y. Wang, Z. Mao, and Y. Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In CVPR, 2021.","title":"Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition","authors":"S Fang, H Xie, Y Wang, Z Mao, Y Zhang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gupta et al., 2016] A. Gupta, A. Vedaldi, and A. Zisserman. Synthetic data for text localisation in natural images. In CVPR, pages 2315-2324, 2016.","title":"Synthetic data for text localisation in natural images","authors":"A Gupta, A Vedaldi, A Zisserman","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2020] W. Hu, X. Cai, J. Hou, S. Yi, and Z. Lin. Gtc: Guided training of ctc towards efficient and accu- rate scene text recognition. In AAAI, pages 11005-11012, 2020.","title":"Gtc: Guided training of ctc towards efficient and accurate scene text recognition","authors":"W Hu, X Cai, J Hou, S Yi, Z Lin","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jaderberg et al., 2014] M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman. Synthetic data and artificial neural networks for natural scene text recognition. In NeurIPS Deep Learning Workshop, 2014.","title":"Synthetic data and artificial neural networks for natural scene text recognition","authors":"M Jaderberg, K Simonyan, A Vedaldi, A Zisserman","year":"2014","venue":"NeurIPS Deep Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jaderberg et al., 2015] M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman. Reading text in the wild with convolutional neural networks. In IJCV, pages 1-20, 2015.","title":"Reading text in the wild with convolutional neural networks","authors":"M Jaderberg, K Simonyan, A Vedaldi, A Zisserman","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karatzas et al., 2015] D. Karatzas, L. Gomez-Bigorda, A. Nicolaou, S. Ghosh, A. Bagdanov, M. Iwamura, J. Matas, L. Neumann, V. R. Chandrasekhar, S. Lu, F. Shafait, S. Uchida, and E. Valveny. Icdar 2015 com- petition on robust reading. In ICDAR, pages 1156-1160, 2015.","title":"Icdar 2015 competition on robust reading","authors":"D Karatzas, L Gomez-Bigorda, A Nicolaou, S Ghosh, A Bagdanov, M Iwamura, J Matas, L Neumann, V R Chandrasekhar, S Lu, F Shafait, S Uchida, E Valveny","year":"2015","venue":"ICDAR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[KaratzasAU et al., 2013] D. KaratzasAU, F. ShafaitAU, S. UchidaAU, M. IwamuraAU, L. G. i. BigordaAU, S. R. MestreAU, J. MasAU, D. F. MotaAU, J. A. AlmazànAU, and L. P. de las Heras. Icdar 2013 robust reading competi- tion. In ICDAR, pages 1484-1493, 2013.","title":"Icdar 2013 robust reading competition","authors":"D Karatzasau, F Shafaitau, S Uchidaau, M Iwamuraau, L G I Bigordaau, S R Mestreau, J Masau, D F Motaau, J A Almazànau, L P De Las Heras","year":"2013","venue":"ICDAR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2019] H. Li, P. Wang, C. Shen, and G. Zhang. Show, attend and read: A simple and strong baseline for ir- regular text recognition. In AAAI, pages 8610-8617, 2019.","title":"Show, attend and read: A simple and strong baseline for irregular text recognition","authors":"H Li, P Wang, C Shen, G Zhang","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luo et al., 2019] C. Luo, L. Jin, and Z. Sun. A multi-object rectified attention network for scene text recognition. In Pattern Recognit., pages 109-118, 2019.","title":"A multi-object rectified attention network for scene text recognition","authors":"C Luo, L Jin, Z Sun","year":"2019","venue":"Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mishra et al., 2012] A. Mishra, A. Karteek, and C. V. Jawa- har. Scene text recognition using higher order language priors. In BMVC, 2012.","title":"Scene text recognition using higher order language priors","authors":"A Mishra, A Karteek, C V Jawahar","year":"2012","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Phan et al., 2013] T. Q. Phan, P. Shivakumara, S. Tian, and C. L. Tan. Recognizing text with perspective distortion in natural scenes. In CVPR, pages 569-576, 2013.","title":"Recognizing text with perspective distortion in natural scenes","authors":"T Q Phan, P Shivakumara, S Tian, C L Tan","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sheng et al., 2019] F. Sheng, Z. Chen, and B. Xu. Nrtr: A no-recurrence sequence-to-sequence model for scene text recognition. In ICDAR, pages 781-786, 2019.","title":"Nrtr: A no-recurrence sequence-to-sequence model for scene text recognition","authors":"F Sheng, Z Chen, B Xu","year":"2019","venue":"ICDAR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shi et al., 2017] B. Shi, X. Bai, and C. Yao. An end-to-end trainable neural network for image-based sequence recog- nition and its application to scene text recognition. In TPAMI, pages 2298-2304, 2017.","title":"An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition","authors":"B Shi, X Bai, C Yao","year":"2017","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shi et al., 2019] B. Shi, M. Yang, X. Wang, P. Lyu, C. Yao, and X. Bai. Aster: An attentional scene text recognizer with flexible rectification. In TPAMI, pages 2035-2048, 2019.","title":"Aster: An attentional scene text recognizer with flexible rectification","authors":"B Shi, M Yang, X Wang, P Lyu, C Yao, X Bai","year":"2019","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tang et al., 2022] X. Tang, Y. Lai, Y. Liu, Y. Fu, and R. Fang. Visual-semantic transformer for scene text recog- nition. In AAAI, 2022.","title":"Visual-semantic transformer for scene text recognition","authors":"X Tang, Y Lai, Y Liu, Y Fu, R Fang","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2011] K. Wang, B. Babenko, and S. Belongie. End-to-end scene text recognition. In ICCV, pages 1457- 1464, 2011.","title":"End-to-end scene text recognition","authors":"K Wang, B Babenko, S Belongie","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2021] Y. Wang, H. Xie, S. Fang, J. Wang, S. Zhu, and Y. Zhang. From two to one: A new scene text recognizer with visual language modeling network. In ICCV, pages 14194-14203, 2021.","title":"From two to one: A new scene text recognizer with visual language modeling network","authors":"Y Wang, H Xie, S Fang, J Wang, S Zhu, Y Zhang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yan et al., 2021] R. Yan, L. Peng, S. Xiao, and G. Yao. Primitive representation learning for scene text recogni- tion. In CVPR, pages 284-293, 2021.","title":"Primitive representation learning for scene text recognition","authors":"R Yan, L Peng, S Xiao, G Yao","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2020] D. Yu, X. Li, C. Zhang, T. Liu, J. Han, J. Liu, and E. Ding. Towards accurate scene text recog- nition with semantic reasoning networks. In CVPR, pages 12113-12122, 2020.","title":"Towards accurate scene text recognition with semantic reasoning networks","authors":"D Yu, X Li, C Zhang, T Liu, J Han, J Liu, E Ding","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhai et al., 2016] C. Zhai, Z. Chen, J. Li, and B. Xu. Chinese image text recognition with blstm-ctc: A segmentation-free method. In CCPR, pages 525-536, 2016.","title":"Chinese image text recognition with blstm-ctc: A segmentation-free method","authors":"C Zhai, Z Chen, J Li, B Xu","year":"2016","venue":"CCPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2020] H. Zhang, Q. Yao, M. Yang, Y. Xu, and X. Bai. Autostr: Efficient backbone search for scene text recognition. In ECCV, pages 751-767. Springer, 2020.","title":"Autostr: Efficient backbone search for scene text recognition","authors":"H Zhang, Q Yao, M Yang, Y Xu, X Bai","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zheng et al., 2021] T. Zheng, Z. Chen, S. Fang, H. Xie, and Y. Jiang. Cdistnet: Perceiving multi-domain character dis- tance for robust text recognition. arXiv:2111.11011, 2021.","title":"Cdistnet: Perceiving multi-domain character distance for robust text recognition","authors":"T Zheng, Z Chen, S Fang, H Xie, Y Jiang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}