{"type":"ScholarlyArticle","title":"Linguistic More: Taking a Further Step toward Efficient and Accurate Scene Text Recognition","authors":[{"name":"Boqiang Zhang","slug":"boqiang-zhang","given":"Boqiang","family":"Zhang","affiliation":"University of Science and Technology of China , Hefei , China"},{"name":"Hongtao Xie","slug":"hongtao-xie","given":"Hongtao","family":"Xie","affiliation":"University of Science and Technology of China , Hefei , China"},{"name":"Yuxin Wang","slug":"yuxin-wang","given":"Yuxin","family":"Wang","affiliation":"University of Science and Technology of China , Hefei , China"},{"name":"Jianjun Xu","slug":"jianjun-xu","given":"Jianjun","family":"Xu","affiliation":"University of Science and Technology of China , Hefei , China"},{"name":"Yongdong Zhang","slug":"yongdong-zhang","given":"Yongdong","family":"Zhang","affiliation":"University of Science and Technology of China , Hefei , China"}],"abstract":"Vision model have gained increasing attention due to their simplicity and efficiency in Scene Text Recognition (STR) task. However, due to lacking the perception of linguistic knowledge and information, recent vision models suffer from two problems: (1) the pure vision-based query results in attention drift, which usually causes poor recognition and is summarized as linguistic insensitive drift (LID) problem in this paper. (2) the visual feature is suboptimal for the recognition in some vision-missing cases (e.g. occlusion, etc.). To address these issues, we propose a Linguistic Perception Vision model (LPV), which explores the linguistic capability of vision model for accurate text recognition. To alleviate the LID problem, we introduce a Cascade Position Attention (CPA) mechanism that obtains high-quality and accurate attention maps through step-wise optimization and linguistic information mining. Furthermore, a Global Linguistic Reconstruction Module (GLRM) is proposed to improve the representation of visual features by perceiving the linguistic information in the visual space, which gradually converts visual features into semantically rich ones during the cascade process. Different from previous methods, our method obtains SOTA results while keeping low complexity (92.4% accuracy with only 8.11M parameters). Code is available at https://github.com/CyrilSterling/LPV.","doi":"10.24963/ijcai.2023/189","url":"https://ijcai.org/proceedings/2023/189","pdf":"https://ijcai.org/proceedings/2023/0189.pdf","markdown":"https://ijcai.org/proceedings/2023/189.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/189","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":189,"pages":{"first":1704,"last":1712},"keywords":["Computer Vision: CV: Recognition (object detection, categorization)","Computer Vision: CV: Scene analysis and understanding","Computer Vision: CV: Vision and language"],"videos":[],"references":[{"position":1,"raw":"[Baek et al., 2019] Jeonghun Baek, Geewook Kim, Jun- yeop Lee, Sungrae Park, Dongyoon Han, Sangdoo Yun, Seong Joon Oh, and Hwalsuk Lee. What is wrong with scene text recognition model comparisons? dataset and model analysis. In International Conference on Computer Vision (ICCV), 2019.","title":"What is wrong with scene text recognition model comparisons? dataset and model analysis","authors":"Baek","year":"2019","venue":"International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cheng et al., 2017] Zhanzhan Cheng, Fan Bai, Yunlu Xu, Gang Zheng, Shiliang Pu, and Shuigeng Zhou. Focusing attention: Towards accurate text recognition in natural im- ages. In Proceedings of the IEEE international conference on computer vision, pages 5076-5084, 2017.","title":"Focusing attention: Towards accurate text recognition in natural images","authors":"Zhanzhan Cheng, Fan Bai, Yunlu Xu, Gang Zheng, Shiliang Pu, Shuigeng Zhou","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Du et al., 2022] Yongkun Du, Zhineng Chen, Caiyan Jia, Xiaoting Yin, Tianlun Zheng, Chenxia Li, Yuning Du, and Yu-Gang Jiang. Svtr: Scene text recognition with a single visual model. arXiv preprint arXiv:2205.00159, 2022.","title":"Svtr: Scene text recognition with a single visual model","authors":"Du","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2022/124"},{"position":6,"raw":"[Fang et al., 2021] Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 7098-7107, 2021.","title":"Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition","authors":"Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, Yongdong Zhang","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Graves et al., 2006] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376, 2006.","title":"Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks","authors":"Alex Graves, Santiago Fernández, Faustino Gomez, Jürgen Schmidhuber","year":"2006","venue":"Proceedings of the 23rd international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gupta et al., 2016] Ankush Gupta, Andrea Vedaldi, and An- drew Zisserman. Synthetic data for text localisation in natural images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2315- 2324, 2016.","title":"Synthetic data for text localisation in natural images","authors":"Ankush Gupta, Andrea Vedaldi, Andrew Zisserman","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2020] Wenyang Hu, Xiaocong Cai, Jun Hou, Shuai Yi, and Zhiping Lin. Gtc: Guided training of ctc towards efficient and accurate scene text recognition. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 11005-11012, 2020.","title":"Gtc: Guided training of ctc towards efficient and accurate scene text recognition","authors":"Hu","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jaderberg et al., 2014] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Synthetic data and artificial neural networks for natural scene text recog- nition. arXiv preprint arXiv:1406.2227, 2014.","title":"Synthetic data and artificial neural networks for natural scene text recognition","authors":"Jaderberg","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jaderberg et al., 2016] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Reading text in the wild with convolutional neural networks. International journal of computer vision, 116(1):1-20, 2016.","title":"Reading text in the wild with convolutional neural networks","authors":"Jaderberg","year":"2016","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karatzas et al., 2013] Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Big- orda, Sergi Robles Mestre, Joan Mas, David Fer- nandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. Icdar 2013 robust reading competition. In 2013 12th international conference on document analysis and recognition, pages 1484-1493. IEEE, 2013.","title":"Icdar 2013 robust reading competition","authors":"Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez I Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, Lluis Pere De, Las Heras","year":"2013","venue":"2013 12th international conference on document analysis and recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Karatzas et al., 2015] Dimosthenis Karatzas, Lluis Gomez- Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bag- danov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. In 2015 13th interna- tional conference on document analysis and recognition (ICDAR), pages 1156-1160. IEEE, 2015.","title":"Icdar 2015 competition on robust reading","authors":"Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu","year":"2015","venue":"2015 13th international conference on document analysis and recognition (ICDAR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Long et al., 2021] Shangbang Long, Xin He, and Cong Yao. Scene text detection and recognition: The deep learning era. International Journal of Computer Vision, 129(1):161-184, 2021.","title":"Shangbang Long, Xin He, and Cong Yao. Scene text detection and recognition: The deep learning era","authors":"Long","year":"2021","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lyu et al., 2019a] Pengyuan Lyu, Zhicheng Yang, Xinhang Leng, Xiaojun Wu, Ruiyu Li, and Xiaoyong Shen. 2d at- tentional irregular scene text recognizer. arXiv preprint arXiv:1906.05708, 2019.","title":"2d attentional irregular scene text recognizer","authors":"Lyu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lyu et al., 2019b] Pengyuan Lyu, Zhicheng Yang, Xinhang Leng, Xiaojun Wu, Ruiyu Li, and Xiaoyong Shen. 2d at- tentional irregular scene text recognizer. arXiv preprint arXiv:1906.05708, 2019.","title":"2d attentional irregular scene text recognizer","authors":"Lyu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mishra et al., 2012] Anand Mishra, Karteek Alahari, and CV Jawahar. Scene text recognition using higher order language priors. In BMVC-British machine vision confer- ence. BMVA, 2012.","title":"Scene text recognition using higher order language priors","authors":"Anand Mishra, Karteek Alahari, Jawahar","year":"2012","venue":"BMVC-British machine vision conference","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Phan et al., 2013] Trung Quy Phan, Palaiahnakote Shivaku- mara, Shangxuan Tian, and Chew Lim Tan. Recognizing text with perspective distortion in natural scenes. In Pro- ceedings of the IEEE International Conference on Com- puter Vision, pages 569-576, 2013.","title":"Recognizing text with perspective distortion in natural scenes","authors":"Trung Quy Phan, Palaiahnakote Shivakumara, Shangxuan Tian, Chew Lim, Tan","year":"2013","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qiao et al., 2020] Zhi Qiao, Yu Zhou, Dongbao Yang, Yu- can Zhou, and Weiping Wang. Seed: Semantics en- hanced encoder-decoder framework for scene text recog- nition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13528- 13537, 2020.","title":"Seed: Semantics enhanced encoder-decoder framework for scene text recognition","authors":"Zhi Qiao, Yu Zhou, Dongbao Yang, Yucan Zhou, Weiping Wang","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Risnumawan et al., 2014] Anhar Risnumawan, Palaia- hankote Shivakumara, Chee Seng Chan, and Chew Lim Tan. A robust arbitrary text detection system for nat- ural scene images. Expert Systems with Applications, 41(18):8027-8048, 2014.","title":"A robust arbitrary text detection system for natural scene images","authors":"Anhar Risnumawan, Palaiahankote Shivakumara, Chee Seng Chan, Chew Lim, Tan","year":"2014","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sheng et al., 2019] Fenfen Sheng, Zhineng Chen, and Bo Xu. Nrtr: A no-recurrence sequence-to-sequence model for scene text recognition. In 2019 International conference on document analysis and recognition (IC- DAR), pages 781-786. IEEE, 2019.","title":"Nrtr: A no-recurrence sequence-to-sequence model for scene text recognition","authors":"Sheng","year":"2019","venue":"2019 International conference on document analysis and recognition (IC-DAR)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shi et al., 2016a] Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence, 39(11):2298-2304, 2016.","title":"An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition","authors":"Shi","year":"2016","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shi et al., 2016b] Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence, 39(11):2298-2304, 2016.","title":"An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition","authors":"Shi","year":"2016","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shi et al., 2018] Baoguang Shi, Mingkun Yang, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai. Aster: An attentional scene text recognizer with flexible rectifica- tion. IEEE transactions on pattern analysis and machine intelligence, 41(9):2035-2048, 2018.","title":"Aster: An attentional scene text recognizer with flexible rectification","authors":"Shi","year":"2018","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wan et al., 2020] Zhaoyi Wan, Minghang He, Haoran Chen, Xiang Bai, and Cong Yao. Textscanner: Reading characters in order for robust scene text recognition. In Proceedings of the AAAI conference on artificial intelli- gence, volume 34, pages 12120-12127, 2020.","title":"Textscanner: Reading characters in order for robust scene text recognition","authors":"Wan","year":"2020","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2011] Kai Wang, Boris Babenko, and Serge Belongie. End-to-end scene text recognition. In 2011 In- ternational conference on computer vision, pages 1457- 1464. IEEE, 2011.","title":"End-to-end scene text recognition","authors":"Kai Wang, Boris Babenko, Serge Belongie","year":"2011","venue":"2011 International conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2021] Yuxin Wang, Hongtao Xie, Shancheng Fang, Jing Wang, Shenggao Zhu, and Yongdong Zhang. From two to one: A new scene text recognizer with vi- sual language modeling network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14194-14203, 2021.","title":"From two to one: A new scene text recognizer with visual language modeling network","authors":"Yuxin Wang, Hongtao Xie, Shancheng Fang, Jing Wang, Shenggao Zhu, Yongdong Zhang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2022a] Peng Wang, Cheng Da, and Cong Yao. Multi-granularity prediction for scene text recognition. In European Conference on Computer Vision, pages 339- 355. Springer, 2022.","title":"Multi-granularity prediction for scene text recognition","authors":"Wang","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2022b] Yuxin Wang, Hongtao Xie, Shancheng Fang, Mengting Xing, Jing Wang, Shenggao Zhu, and Yongdong Zhang. Petr: Rethinking the capability of transformer-based language model in scene text recogni- tion. IEEE Transactions on Image Processing, 31:5585- 5598, 2022.","title":"Petr: Rethinking the capability of transformer-based language model in scene text recognition","authors":"Wang","year":"2022","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xu et al., 2022] Jianjun Xu, Hongtao Xie, Hai Xu, Yuxin Wang, Sun-ao Liu, and Yongdong Zhang. Boat in the sky: Background decoupling and object-aware pooling for weakly supervised semantic segmentation. In Proceedings of the 30th ACM International Conference on Multimedia, pages 5783-5792, 2022.","title":"Boat in the sky: Background decoupling and object-aware pooling for weakly supervised semantic segmentation","authors":"Jianjun Xu, Hongtao Xie, Hai Xu, Yuxin Wang, Sun-Ao Liu, Yongdong Zhang","year":"2022","venue":"Proceedings of the 30th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yu et al., 2020] Deli Yu, Xuan Li, Chengquan Zhang, Tao Liu, Junyu Han, Jingtuo Liu, and Errui Ding. Towards ac- curate scene text recognition with semantic reasoning net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12113- 12122, 2020.","title":"Towards accurate scene text recognition with semantic reasoning networks","authors":"Deli Yu, Xuan Li, Chengquan Zhang, Tao Liu, Junyu Han, Jingtuo Liu, Errui Ding","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yue et al., 2020] Xiaoyu Yue, Zhanghui Kuang, Chenhao Lin, Hongbin Sun, and Wayne Zhang. Robustscanner: Dynamically enhancing positional clues for robust text recognition. In European Conference on Computer Vision, pages 135-151. Springer, 2020.","title":"Robustscanner: Dynamically enhancing positional clues for robust text recognition","authors":"Xiaoyu Yue, Zhanghui Kuang, Chenhao Lin, Hongbin Sun, Wayne Zhang","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zheng et al., 2021] Tianlun Zheng, Zhineng Chen, Shancheng Fang, Hongtao Xie, and Yu-Gang Jiang. Cdistnet: Perceiving multi-domain character distance for robust text recognition. arXiv preprint arXiv:2111.11011, 2021.","title":"Cdistnet: Perceiving multi-domain character distance for robust text recognition","authors":"Tianlun Zheng, Zhineng Chen, Shancheng Fang, Hongtao Xie, Yu-Gang Jiang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}