{"type":"ScholarlyArticle","title":"Learning to Read Irregular Text with Attention Mechanisms","authors":[{"name":"Xiao Yang","slug":"xiao-yang","given":"Xiao","family":"Yang","affiliation":"The Pennsylvania State University , University Park , PA 16802 , USA"},{"name":"Dafang He","slug":"dafang-he","given":"Dafang","family":"He","affiliation":"The Pennsylvania State University , University Park , PA 16802 , USA"},{"name":"Zihan Zhou","slug":"zihan-zhou","given":"Zihan","family":"Zhou","affiliation":"The Pennsylvania State University , University Park , PA 16802 , USA"},{"name":"Daniel Kifer","slug":"daniel-kifer","given":"Daniel","family":"Kifer","affiliation":"The Pennsylvania State University , University Park , PA 16802 , USA"},{"name":"C. Lee Giles","slug":"c-lee-giles","given":"C. Lee","family":"Giles","affiliation":"The Pennsylvania State University , University Park , PA 16802 , USA"}],"abstract":"We present a robust end-to-end neural-based model to attentively recognize text in natural images. Particularly, we focus on accurately identifying irregular (perspectively distorted or curved) text, which has not been well addressed in the previous literature. Previous research on text reading often works with regular (horizontal and frontal) text and does not adequately generalize to processing text with perspective distortion or curving effects. Our work proposes to overcome this difficulty by introducing two learning components: (1) an auxiliary dense character detection task that helps to learn text specific visual patterns, (2) an alignment loss that provides guidance to the training of an attention model. We show with experiments that these two components are crucial for achieving fast convergence and high classification accuracy for irregular text recognition. Our model outperforms previous work on two irregular-text datasets: SVT-Perspective and CUTE80, and is also highly-competitive on several regular-text datasets containing primarily horizontal and frontal text.","doi":"10.24963/ijcai.2017/458","url":"https://ijcai.org/proceedings/2017/458","pdf":"https://ijcai.org/proceedings/2017/0458.pdf","markdown":"https://ijcai.org/proceedings/2017/458.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/458","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":458,"pages":{"first":3280,"last":3286},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning","Robotics and Vision: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Bissacco et al., 2013] Alessandro Bissacco, Mark Cum- mins, Yuval Netzer, and Hartmut Neven. Photoocr: Read- ing text in uncontrolled conditions. In ICCV, 2013.","title":"Photoocr: Reading text in uncontrolled conditions","authors":"Alessandro Bissacco, Mark Cummins, Yuval Netzer, Hartmut Neven","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. arXiv preprint arXiv:1409.1259, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goodfellow et al., 2013] Ian Goodfellow, David Warde- farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In ICML, 2013.","title":"Maxout networks","authors":"Ian Goodfellow, David Wardefarley, Mehdi Mirza, Aaron Courville, Yoshua Bengio","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Graves et al., 2006] Alex Graves, Santiago Fernández, Faustino J Gomez, and Jürgen Schmidhuber. Connec- tionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks. In ICML, 2006.","title":"Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks","authors":"Alex Graves, Santiago Fernández, Faustino J Gomez, Jürgen Schmidhuber","year":"2006","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gupta et al., 2016] Ankush Gupta, Andrea Vedaldi, and An- drew Zisserman. Synthetic data for text localisation in nat- ural images. In CVPR, 2016.","title":"Synthetic data for text localisation in natural images","authors":"Ankush Gupta, Andrea Vedaldi, Andrew Zisserman","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Pan He, Weilin Huang, Yu Qiao, Chen Change Loy, and Xiaoou Tang. Reading scene text in deep convolutional sequences. In AAAI, 2016.","title":"Reading scene text in deep convolutional sequences","authors":"He","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jaderberg et al., 2014a] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Synthetic data and artificial neural networks for natural scene text recog- nition. In NIPS Workshop, 2014.","title":"Synthetic data and artificial neural networks for natural scene text recognition","authors":"Jaderberg","year":"2014","venue":"NIPS Workshop","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jaderberg et al., 2014b] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Deep features for text spotting. In ECCV. Springer, 2014.","title":"Deep features for text spotting","authors":"Jaderberg","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jaderberg et al., 2016] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Reading text in the wild with convolutional neural networks. IJCV, 116, 2016.","title":"Reading text in the wild with convolutional neural networks","authors":"Jaderberg","year":"2016","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Julien et al., 2011] Rabin Julien, Gabriel Peyré, Julie De- lon, and Bernot Marc. Wasserstein barycenter and its ap- plication to texture mixing. In SSVM. Springer, 2011.","title":"Wasserstein barycenter and its application to texture mixing","authors":"Rabin Julien, Gabriel Peyré, Julie Delon, Bernot Marc","year":"2011","venue":"SSVM","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee and Osindero, 2016] Chen-Yu Lee and Simon Osin- dero. Recursive recurrent nets with attention modeling for ocr in the wild. In CVPR. IEEE, 2016.","title":"Recursive recurrent nets with attention modeling for ocr in the wild","authors":"Chen-Yu Lee, Simon Osindero","year":"2016","venue":"CVPR. IEEE","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2016] Yi Li, Kaiming He, Jian Sun, et al. R-fcn: Object detection via region-based fully convolutional net- works. In NIPS, 2016.","title":"R-fcn: Object detection via region-based fully convolutional networks","authors":"Yi Li, Kaiming He, Jian Sun","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In CVPR, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lucas et al., 2003] Simon M Lucas, Alex Panaretos, Luis Sosa, Anthony Tang, Shirley Wong, and Robert Young. Icdar 2003 robust reading competitions. In ICDAR, 2003.","title":"Icdar 2003 robust reading competitions","authors":"Alex Simon M Lucas, Luis Panaretos, Anthony Sosa, Shirley Tang, Robert Wong, Young","year":"2003","venue":"ICDAR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mishra et al., 2012] Anand Mishra, Karteek Alahari, and CV Jawahar. Scene text recognition using higher order language priors. In BMVC, 2012.","title":"Scene text recognition using higher order language priors","authors":"Anand Mishra, Karteek Alahari, Jawahar","year":"2012","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Neumann and Matas, 2012] Lukáš Neumann and Jiří Matas. Real-time scene text localization and recognition. In CVPR. IEEE, 2012.","title":"Real-time scene text localization and recognition","authors":"Matas Neumann, Lukáš Neumann, Jiří Matas","year":"2012","venue":"CVPR. IEEE","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ni et al., 2009] Kangyu Ni, Xavier Bresson, Tony Chan, and Selim Esedoglu. Local histogram based segmentation using the wasserstein distance. IJCV, 84, 2009.","title":"Local histogram based segmentation using the wasserstein distance","authors":"Ni","year":"2009","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Noh et al., 2015] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for se- mantic segmentation. In ICCV. IEEE, 2015.","title":"Learning deconvolution network for semantic segmentation","authors":"Noh","year":"2015","venue":"ICCV. IEEE","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pinheiro et al., 2016] Pedro O Pinheiro, Tsung-Yi Lin, Ro- nan Collobert, and Piotr Dollár. Learning to refine object segments. In ECCV. Springer, 2016.","title":"Learning to refine object segments","authors":"Tsung-Yi Pedro O Pinheiro, Ronan Lin, Piotr Collobert, Dollár","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Quy Phan et al., 2013] Trung Quy Phan, Palaiahnakote Shivakumara, Shangxuan Tian, and Chew Lim Tan. Rec- ognizing text with perspective distortion in natural scenes. In ICCV, 2013.","title":"Recognizing text with perspective distortion in natural scenes","authors":"Trung Quy Phan, Palaiahnakote Shivakumara, Shangxuan Tian, Chew Lim, Tan","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Risnumawan et al., 2014] Anhar Risnumawan, Palaia- hankote Shivakumara, Chee Seng Chan, and Chew Lim Tan. A robust arbitrary text detection system for natural scene images. Expert Systems with Applications, 41, 2014.","title":"A robust arbitrary text detection system for natural scene images","authors":"Anhar Risnumawan, Palaiahankote Shivakumara, Chee Seng Chan, Chew Lim, Tan","year":"2014","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shi et al., 2016a] Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. CVPR, 2016.","title":"An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition","authors":"Shi","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shi et al., 2016b] Baoguang Shi, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai. Robust scene text recognition with automatic rectification. In CVPR. IEEE, 2016.","title":"Robust scene text recognition with automatic rectification","authors":"Shi","year":"2016","venue":"CVPR. IEEE","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2011] Kai Wang, Boris Babenko, and Serge Belongie. End-to-end scene text recognition. In ICCV. IEEE, 2011.","title":"End-to-end scene text recognition","authors":"Kai Wang, Boris Babenko, Serge Belongie","year":"2011","venue":"ICCV. IEEE","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2012] Tao Wang, David J Wu, Adam Coates, and Andrew Y Ng. End-to-end text recognition with con- volutional neural networks. In ICPR. IEEE, 2012.","title":"End-to-end text recognition with convolutional neural networks","authors":"Tao Wang, David J Wu, Adam Coates, Andrew Y Ng","year":"2012","venue":"ICPR. IEEE","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdi- nov, Richard S Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual at- tention. In ICML, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yao et al., 2014] Cong Yao, Xiang Bai, Baoguang Shi, and Wenyu Liu. Strokelets: A learned multi-scale representa- tion for scene text recognition. In CVPR, 2014.","title":"Strokelets: A learned multi-scale representation for scene text recognition","authors":"Cong Yao, Xiang Bai, Baoguang Shi, Wenyu Liu","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zeiler, 2012] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.","title":"Adadelta: an adaptive learning rate method","authors":"D Matthew, Zeiler","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu et al., 2006] Qiang Zhu, Shai Avidan, Meichen Yeh, and Tim Kwang-Ting Cheng. Fast human detection using a cascade of histograms of oriented gradients. In CVPR, 2006.","title":"Fast human detection using a cascade of histograms of oriented gradients","authors":"Qiang Zhu, Shai Avidan, Meichen Yeh, Tim Kwang-Ting Cheng","year":"2006","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}