{"type":"ScholarlyArticle","title":"Learning to Draw Text in Natural Images with Conditional Adversarial Networks","authors":[{"name":"Shancheng Fang","slug":"shancheng-fang","given":"Shancheng","family":"Fang","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences; School of Cyber Security , University of Chinese Academy of Sciences"},{"name":"Hongtao Xie","slug":"hongtao-xie","given":"Hongtao","family":"Xie","affiliation":"School of Information Science and Technology , University of Science and Technology of China"},{"name":"Jianjun Chen","slug":"jianjun-chen","given":"Jianjun","family":"Chen","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences"},{"name":"Jianlong Tan","slug":"jianlong-tan","given":"Jianlong","family":"Tan","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences"},{"name":"Yongdong Zhang","slug":"yongdong-zhang","given":"Yongdong","family":"Zhang","affiliation":"School of Information Science and Technology , University of Science and Technology of China"}],"abstract":"In this work, we propose an entirely learning-based method to automatically synthesize text sequence in natural images leveraging conditional adversarial networks. As vanilla GANs are clumsy to capture structural text patterns, directly employing GANs for text image synthesis typically results in illegible images. Therefore, we design a two-stage architecture to generate repeated characters in images. Firstly, a character generator attempts to synthesize local character appearance independently, so that the legible characters in sequence can be obtained. To achieve style consistency of characters, we propose a novel style loss based on variance-minimization. Secondly, we design a pixel-manipulation word generator constrained by self-regularization, which learns to convert local characters to plausible word image. Experiments on SVHN dataset and ICDAR, IIIT5K datasets demonstrate our method is able to synthesize visually appealing text images. Besides, we also show the high-quality images synthesized by our method can be used to boost the performance of a scene text recognition algorithm.","doi":"10.24963/ijcai.2019/101","url":"https://ijcai.org/proceedings/2019/101","pdf":"https://ijcai.org/proceedings/2019/0101.pdf","markdown":"https://ijcai.org/proceedings/2019/101.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/101","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":101,"pages":{"first":715,"last":722},"keywords":["Computer Vision: Statistical Methods and Machine Learning","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Azadi et al., 2018] Samaneh Azadi, Matthew Fisher, Vladimir G Kim, Zhaowen Wang, Eli Shechtman, and Trevor Darrell. Multi-content gan for few-shot font style transfer. In CVPR, pages 7564-7573, 2018.","title":"Multi-content gan for few-shot font style transfer","authors":"Azadi","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brock et al., 2019] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2019.","title":"Large scale gan training for high fidelity natural image synthesis","authors":"Andrew Brock, Jeff Donahue, Karen Simonyan","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dosovitskiy and Brox, 2016] Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In NIPS, 2016.","title":"Generating images with perceptual similarity metrics based on deep networks","authors":"Brox Dosovitskiy, Alexey Dosovitskiy, Thomas Brox","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dumoulin et al., 2017] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In ICLR, 2017.","title":"A learned representation for artistic style","authors":"Jonathon Vincent Dumoulin, Manjunath Shlens, Kudlur","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fang et al., 2018] Shancheng Fang, Hongtao Xie, Zheng- Jun Zha, Nannan Sun, Jianlong Tan, and Yongdong Zhang. Attention and language ensemble for scene text recogni- tion with convolutional sequence modeling. In ACM MM, pages 248-256, 2018.","title":"Attention and language ensemble for scene text recognition with convolutional sequence modeling","authors":"Shancheng Fang, Hongtao Xie, Zheng-Jun Zha, Nannan Sun, Jianlong Tan, Yongdong Zhang","year":"2018","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gatys et al., 2016] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR, pages 2414-2423, 2016.","title":"Image style transfer using convolutional neural networks","authors":"Leon A Gatys, Alexander S Ecker, Matthias Bethge","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gupta et al., 2016] Ankush Gupta, Andrea Vedaldi, and An- drew Zisserman. Synthetic data for text localisation in nat- ural images. In CVPR, pages 2315-2324, 2016.","title":"Synthetic data for text localisation in natural images","authors":"Ankush Gupta, Andrea Vedaldi, Andrew Zisserman","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS, 2017.","title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In CVPR, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jaderberg et al., 2014] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Synthetic data and artificial neural networks for natural scene text recog- nition. In NIPS Deep Learning Workshop, 2014.","title":"Synthetic data and artificial neural networks for natural scene text recognition","authors":"Jaderberg","year":"2014","venue":"NIPS Deep Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ledig et al., 2017] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Cunningham, et al. Photo-realistic single image super-resolution using a generative adversar- ial network. In CVPR, pages 4681-4690, 2017.","title":"Photo-realistic single image super-resolution using a generative adversarial network","authors":"Ledig","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li and Wand, 2016] Chuan Li and Michael Wand. Precom- puted real-time texture synthesis with markovian genera- tive adversarial networks. In ECCV, pages 702-716, 2016.","title":"Precomputed real-time texture synthesis with markovian generative adversarial networks","authors":"Wand ; ] Chuan Li, Michael Li, Wand","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2018] Yang Liu, Zhaowen Wang, Hailin Jin, and Ian Wassell. Synthetically supervised feature learning for scene text recognition. In ECCV, pages 435-451, 2018.","title":"Synthetically supervised feature learning for scene text recognition","authors":"Yang Liu, Zhaowen Wang, Hailin Jin, Ian Wassell","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lucas et al., 2003] Simon M Lucas, Alex Panaretos, Luis Sosa, Anthony Tang, Shirley Wong, and Robert Young. Icdar 2003 robust reading competitions. In ICDAR, 2003.","title":"Icdar 2003 robust reading competitions","authors":"Alex Simon M Lucas, Luis Panaretos, Anthony Sosa, Shirley Tang, Robert Wong, Young","year":"2003","venue":"ICDAR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mishra et al., 2012] Anand Mishra, Karteek Alahari, and CV Jawahar. Scene text recognition using higher order language priors. In BMVC, 2012.","title":"Scene text recognition using higher order language priors","authors":"Anand Mishra, Karteek Alahari, Jawahar","year":"2012","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Miyato and Koyama, 2018] Takeru Miyato and Masanori Koyama. cgans with projection discriminator. In ICLR, 2018.","title":"cgans with projection discriminator","authors":"Koyama ; ] Takeru Miyato, Masanori Miyato, Koyama","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Miyato et al., 2018] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, et al. Spectral normalization for generative adversarial networks. In ICLR, 2018.","title":"Spectral normalization for generative adversarial networks","authors":"Miyato","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Netzer et al., 2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learn- ing. In NIPS workshop, volume 2011, page 5, 2011.","title":"Reading digits in natural images with unsupervised feature learning","authors":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y Ng","year":"2011","venue":"NIPS workshop","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peters et al., 2018] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018.","title":"Deep contextualized word representations","authors":"Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer","year":"2018","venue":"Proc. of NAACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Reed et al., 2016] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In ICML, pages 1060-1069, 2016.","title":"Generative adversarial text to image synthesis","authors":"Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Salimans et al., 2016] Tim Salimans, Ian Goodfellow, Woj- ciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NIPS, 2016.","title":"Improved techniques for training gans","authors":"Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shrivastava et al., 2017] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In CVPR, 2017.","title":"Learning from simulated and unsupervised images through adversarial training","authors":"Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, Russell Webb","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pages 2818-2826, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tran et al., 2017] Dustin Tran, Rajesh Ranganath, and David M. Blei. Hierarchical implicit models and likelihood-free variational inference. In NIPS, 2017.","title":"Hierarchical implicit models and likelihood-free variational inference","authors":"Dustin Tran, Rajesh Ranganath, David M Blei","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vedaldi, 2016] Victor Lempitsky Dmitry Ulyanov Andrea Vedaldi. Instance normalization: The missing ingredient for fast stylization. arXiv preprint:1607.08022, 2016.","title":"Instance normalization: The missing ingredient for fast stylization","authors":"Victor Lempitsky, Dmitry Ulyanov, Andrea Vedaldi","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xie et al., 2019] Hongtao Xie, Shancheng Fang, Zheng-Jun Zha, Yating Yang, Yan Li, et al. Convolutional attention networks for scene text recognition. ACM TOMM, 2019.","title":"Convolutional attention networks for scene text recognition","authors":"Hongtao Xie, Shancheng Fang, Zheng-Jun Zha, Yating Yang, Yan Li","year":"2019","venue":"ACM TOMM","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhan et al., 2018] Fangneng Zhan, Shijian Lu, and Chuhui Xue. Verisimilar image synthesis for accurate detection and recognition of texts in scenes. In ECCV, 2018.","title":"Verisimilar image synthesis for accurate detection and recognition of texts in scenes","authors":"Zhan","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2018] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint:1805.08318, 2018.","title":"Self-attention generative adversarial networks","authors":"Han Zhang, Ian Goodfellow, Dimitris Metaxas, Augustus Odena","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}