Learning to Draw Text in Natural Images with Conditional Adversarial Networks

1 Introduction

Drawing text can be considered as a problem of image synthesis that focuses on text rendering. We humans are able to write or design text, practicing in communication, typography, hand-lettering, etc. However, it remains a challenge for artificial intelligence to manipulate the probability distributions of text images in uncontrolled conditions. In this work, we explore an approach to automatically synthesize text sequence in natural images given the text labels.

A straightforward way to synthesize text images is applying the non-learning based method [Jaderberg et al., 2014; Gupta et al., 2016; Zhan et al., 2018]. By setting several parameters in advance, such as font, color, distortion, noise, etc. [Jaderberg et al., 2014], this kind of method has the ability to semi-automatically render images with diverse text using the techniques of computer graphics. However, it is tedious for a non-learning approach to appropriately select the parameters and realistically simulate natural images (uneven lighting, occlusion, degradation, etc.). In addition, it also fails to synthesize images with unseen font, e.g., hand-writing and special-effects text.

* Hongtao Xie (htxie@ustc.edu.cn)

To better simulate natural scenes and produce realistic text images, the learning based method can be considered. Recently, image synthesis has advanced dramatically with the emergence of generative adversarial networks (GANs) [Goodfellow et al., 2014]. Though GANs show impressive results in high-quality image generation, it is still an intractable problem to capture geometric or structural scenes [Zhang et al., 2018; Ledig et al., 2017]. As characters occur repeatedly in text string, there are strong structural patterns in text images. Therefore, directly employing vanilla GANs for text image synthesis typically recovers unreadable or meaningless images (Figure 7a). Specifically, applying learning based method for text image synthesis confronts the following challenges: firstly, the text images should be generated in arbitrary length. Secondly, compared to the imageto-image translation task [Isola et al., 2017] that requires an image as input, synthesizing text images from scratch only takes character labels, which carry less information than images. Besides, different from object synthesis that generally one salient category is placed in the images, characters in text sequence are of the same importance. Thus, how to keep all characters readable is the primary issue. In addition, characters possess dependency with each other and tend to share similar styles in natural images.

Taking the above challenges into consideration, we propose an entirely learning-based method leveraging conditional adversarial networks, called scene text synthesis GAN (STS-GAN). Inspired by the learning process that humans master characters writing first, and then are capable of drawing text freely, we divide the generation procedure into character synthesis and word synthesis stages. Figure 1 illustrates the framework of STS-GAN. In the former stage, a character generator attempts to capture local character appearance based on conditional GAN. The character generator separately synthesizes character images, so that the legible characters in sequence can be obtained. To resolve the problem of style discrepancy, we propose a novel style loss based on variance-minimization. The style loss aims to minimize the characters distance in style space, enabling characters in the same text patch to share similar styles (e.g., font and background). In the latter stage, we design a pixelmanipulation word generator conditioned on text labels to produce variable-length text images. Training with adversarial learning and self-regularization, the word generator is able to learn the characters dependency in natural scenes, and it can consequently map the synthetic character images to perceptually convincing word image. To the best of our knowledge, our method is the first one to synthesize text images from scratch entirely based on learning. The experiments conducted on SVHN dataset and ICDAR, IIIT5K datasets show that STS-GAN has the ability to synthesize high-quality text images within a complex environment. In addition, compared to the non-learning methods, our STS-GAN can better fit the desired data distribution in natural scene images. Specifically, the main contributions of this paper include: (1) we construct STS-GAN to synthesize text in natural images, which is an entirely learning-based model using welldesigned conditional GANs. The STS-GAN is designed as a novel two-stage architecture aiming to capture structural patterns in text images. (2) A novel style loss based on feature variance minimization is proposed to generate character images in harmonious patterns. (3) Experiments on challenging scene text datasets show STS-GAN is able to synthesize highly deceptive text images. In addition, the synthetic images can be used to boost the performance of a text recognition algorithm.

2 Background

The generative adversarial networks (GANs) aim to learn a mapping function G from random noise vector z to image x [Goodfellow et al., 2014]. The mapping function G : z → x, named generator, is adversarially trained with a discriminator D through the following objective function:

where x′ is the real image. G tries to minimize this objective while D tries to maximize it, i.e., G∗ = arg min G max D LGAN(G,D). Based on Equation 1, conditional GANs are proposed to control the generation by injecting supervised information y into both G and D. The formulas below are a hinge based method [Tran et al., 2017] to train G and D iteratively:

where y typically could be labels in class-conditional image synthesis.

Showing promising performance in image synthesis, GANs have been further improved to impart stable training and high-quality images by many recent works. For example, to stabilize the training, Miyato et al. [2018] introduce spectral normalization (SN) that constrains the Lipschitz constant of the discriminator. Further, projection discriminator [Miyato and Koyama, 2018] is proposed taking an inner product between the embedded class vector and the feature vector, which improves class-conditional performance greatly. Other works attempt to explore helpful tricks for GANs, such as BigGANs [Brock et al., 2019] which trains networks with larger scale (network channels and training batch), achieving impressive results on ImageNet.

The powerful capability makes GANs competent for many specific tasks. Ledig et al. [2017] train a GAN model combining with traditional content loss for image superresolution. In [Isola et al., 2017], an image-conditional model, called Pix2Pix, is proposed for image-to-image translation problems. Also based on conditional GANs, Reed et al. [2016] develop GAN-CLS to synthesize images given text descriptions. These methods above mainly focus on generating images with few structural scenes. MC-GAN [Azadi et al.,

2018] pays attention to designing glyphs, which first generates glyph then transfers color and ornamentation. Nevertheless, MC-GAN synthesizes each character images independently without sequence information. In [Liu et al., 2018], canonical rendering process is applied to synthesize text images and the adversarial loss is introduced for feature learning, which is also deviated from our method that synthesizes text images from scratch.

3 Scene Text Synthesis GAN

Mathematically, we denote the characters collection as C = {yi}Ni=1 , where N is the number of characters. Given a text vector y ∈ CM , where M is text length, our goal is to generate corresponding image collection X, that each image x ∈ X is a readable representation of y in natural image. Figure 1 depicts the pipeline of our approach. To generate an image that contains characters sequence y, we divide this procedure into two stages: character synthesis and word synthesis. In the following sections, we will detail our approach from the aspects of character synthesis and word synthesis.

3.1 Character Synthesis

Character synthesis stage describes the skeleton for a text image. Taking character label y ∈ C and latent vector z as input, this stage samples character image xc by directly applying the conditional GAN. The scale of xc is set to 32 × 32 pixels.

Variance-Minimization Style Loss

Even though the conditional GAN is able to generate legible characters as expected, it is impractical to integrate all the character images into text patch due to the high diversity of styles between each character (Figure 4a). Therefore, we try to explore a method to unify the style of each character in the same image. Within our framework, for a fixed latent vector z, the output images of different y ∈ C should visually be consistent in font, color, background, etc. However, the conditional GANs are powerless for this and the style of xc ∈ {Gc(z, y)|y ∈ C} is out of control.

Suppose S ∈ Rn is a style space that closer points in S have similar styles. Given a mapping function ψ(·) that projects xc to corresponding representation in S, we can apply ψ(·) to guide the training of Gc by restricting the distances of synthetic images in S, so that Gc can generate character images with consistent style. For all the synthetic images xc ∈ {Gc(z, y)|y ∈ C}, minimizing the variance of corresponding representations in S means reducing the style discrepancy. Based on this observation, we propose the following objective:

The style function ψ(·) can be parametrized as a neural network. We share the network of ψ(·) with discriminator Dc, as it adds negligible computation to GAN training. Thus, Equation 4 can be rewritten as:

We note that Equation 5 can be easily approximated with Monte Carlo simulation. For the estimation of V ary, we sample 8 labels for each latent vector z. Then the mean value of each feature dimension in S is calculated after variance estimation. Therefore, Equation 5 can be optimized by traditional stochastic gradient descent together with GAN loss.

Table 1: Network architectures for character model. chi is the channel width of ResBlock.

In summary, combining conditional adversarial loss and style loss, our training objectives for Dc and Gc are concluded as follows:

where λs is a balance parameter between the GAN loss and style loss.

Our style loss is related to perceptual loss [Dosovitskiy and Brox, 2016; Gatys et al., 2016] as both of us employ an additional network for feature space projection. However, our method differs from perceptual loss as we attempt to reduce feature variance between synthetic images from the same latent vector z, rather than minimizing the norm of feature vectors between real and fake images.

Network Architecture

Our network architectures are designed mainly following [Miyato et al., 2018; Miyato and Koyama, 2018], and the details are given in Table 1. Overall, we employ ResBlock defined in [Miyato and Koyama, 2018] as the backbone of both generator and discriminator. Spectral normalization is applied to each parameter layer both in generator and discriminator. For generator Gc, class label y is injected using classconditional batch normalization [Dumoulin et al., 2017]. For discriminator Dc, y is provided to Dc based on projection [Miyato and Koyama, 2018]. In addition, the style function in Equation 5 shares the network with discriminator Dc. We extract the output of global sum pooling in Table 1b as the style representation, so that D′c (Gc(z, y)) ∈ Rch1 , where ch1 is the channel width of the last ResBlock in discriminator network.

3.2 Word Synthesis

By now we can synthesize character sequence in images by simply applying image stitching based on the fruit of Section 3.1. However, in this case there exist seams and discontinuities between characters, causing pool visual effects (top rows in Figure 6). In the word synthesis stage, we aim to fine-tune the stitched image ˆxw to more visually appealing image xw. The stitched image ˆxw, whose aspect ratio is kept invariant, i.e. the size is 32 × (M · 32), is fed into word model together with its character labels y ∈ CM . We also design the word model following the idea of conditional GANs, in which the word generator is a U-Net [Ronneberger et al., 2015] implementing pixel-wise manipulation.

Training Objective

To refine ˆxw to xw, two aspects should be taken into account. The first is how to retain text structure in ˆxw. The second is how to remove noise and achieve realism, i.e., removing discontinuities and rearranging the characters layout. To preserve local appearance, we introduce L1 loss [Isola et al., 2017; Shrivastava et al., 2017] for self-regularization:

To generate realistic images, we turn to a variant of conditional GAN. Different from character synthesis that applies character label y to Equation 2,3, we use word label y to control the content instead. Therefore, Dw and Gw can be optimized by minimizing the following objectives:

where xw = Gw(ˆxw, y) and λl is a balance parameter between the GAN loss and L1 loss.

As GANs tend to disrupt the text structure obtained from ˆxw, the L1 self-regularization is crucial to preserve character appearance. Compared to generating text image from scratch, pixel-wise manipulation with self-regularization substantially reduces the underlying sample space, which eases the burden on GAN synthesis. Different from the L1 loss in [Isola et al., 2017] applied between real and fake images, we restrict the output of the generator to its input. Our method also deviates from [Shrivastava et al., 2017], as we explicitly consider word label y both in generator and discriminator.

Pixel-wise Manipulation and Patch Adversarial Learning

A basic demand of word model is to deal with variable-length images as the size of ˆxw is 32 × (M · 32). Therefore, we implement both generator Gw and discriminator Dw as fully convolutional networks. For the generator Gw, we design the network architecture following U-Net, allowing Gw to manipulate images from pixels to pixels. Table 2 details the architecture, where conv and deconv are convolution and transposed convolution both with kernel 4 × 4 and stride 2; IN is instance normalization [Vedaldi, 2016] conditioned on y. The feature maps from layer i and n - i are concatenated, and then fed to layer n - i + 1, where n = 10 is the total number of layers.

As a strong discriminator is substantially important for providing correct gradient to generator training, we exploit several schemes to build Dw. First, Dw adopts the backbone of Dc (Table 1b) as its network, thus Dw can be initialized with well-trained Dc, which enables Dw to distinguish text patch initially. More concretely, we drop global sum pooling in Table 1b, resulting in an output response r ∈ R8×(M·8) , and the fully-connected (fc) layer is replaced by a 1 × 1 convolution layer. Note that fc layer can be viewed as 1 × 1 convolution operated on feature maps with shape 1 × 1, so that the 1 × 1 convolution layer in Dw can be initialized with fc layer in Dc. Then, we employ adversarial loss to local image patches [Li and Wand, 2016] rather than the full image, as it helps to keep local character structure. In our method, the local image patches are at the scale of 48×48 (if available), which is the size of receptive field in r. We apply Equation 9 to each element in r, and average the results to get the final discriminator response.

To inject labels into Gw and Dw, a straightforward way is assigning the label of each pixel in input image to its corresponding location in the feature map. More formally, pi,j is a pixel in fake image xw or real image x′w , and s is the stride from image to feature map f ∈ RC×(32/s)×(32·W/s) , where C is the number of channels. Thus, p′i/s,j/s in f has the same label y with pi,j. Besides, the labels in xw and x′w are easily obtained according to the image stitching and annotations, respectively. After determining the labels for feature maps, class-conditional instance normalization in generator and projection in discriminator can be conducted.

4 Experiments

4.1 Datasets and EvaluationMetrics

Datasets

We evaluate the proposed STS-GAN on two scene text datasets. The first is Street View House Number (SVHN) [Netzer et al., 2011], which contains 10 classes from digit 1 to 10. There are 73257 training and 26032 test character images, and 33402 training and 13068 test word images in SVHN dataset. The second is an alphanumeric (62 characters) dataset composed of data from ICDAR 2003 [Lucas et al., 2003] and IIIT 5K-word [Mishra et al., 2012] datasets. This dataset (IC03+IIIT) has 15791 training and 20648 test character images. We keep the images with no more than twenty characters, thus the number of training and test word images are 3096 and 4056, respectively.

Evaluation Metrics

We primarily apply Inception score (IS) [Salimans et al., 2016] to quantitatively evaluate the synthetic text images, as a high score generally indicates better visual appearance. In addition, Fr´echet Inception distance (FID) [Heusel et al., 2017] is used as an assistant tool to assess the realism and variation. Different from IS, lower FID means better. Rather than employing Inception network [Szegedy et al., 2016] trained on ImageNet, we use ResNet classifiers [He et al., 2016] trained on SVHN or IC03+IIIT datasets for IS and FID. Specifically, the ResNet classifiers are originally designed for CIFAR-10 with 56 layers. For character classifier, the size of input image is 32 × 32, and the feature vector before the last fullyconnected layer is adopted to calculate the FID. For word classifier, the size of input image is 32 × 128. We share the backbone of character classifier with word classifier, which is equipped with an LSTM layer trained with CTC loss. The feature vector used for FID is the output of the last residual block. In all experiments, 50k samples are randomly sampled to compute the IS and FID scores for each evaluation.

Implementation Details

We employ Adam optimizer as solver with momentum β1 = 0 and β2 = 0.999 for all the networks. The learning rates are 2 · 10-4 for Dc and 5 · 10-5 for Gc, and 2 · 10-4 for both Dw and Gw. For character model, batch size is set to 512. For word model, we sample 8 images with the same word length each batch. All the latent vector z is sampled from standard Gaussian distribution.

4.2 Evaluation on SVHN

To validate the effectiveness of STS-GAN, we first conduct experiments on a digit dataset. The margins of both character and word images are removed, thus the text is center-cropped in the images. Figure 2 presents some synthetic text images (top rows). Compared to the real images (bottom row), the fake images with arbitrary text length are highly deceptive.

Figure 3 plots the IS and FID training curves for character model (top) and word model (bottom). On one hand, the converged curves indicate STS-GAN is able to synthesize better images with increasing iterations. On the other hand, they also prove that by directly transferring IS and FID to our tasks, IS and FID can be effective metrics to evaluate the generation of text images, even though the character classifier and word classifier are newly trained on text image datasets.

4.3 Evaluation on ICDAR and IIIT5K

To have a deep insight into STS-GAN, we further conduct experiments on a more challenging IC03+IIIT dataset. Compared to SVHN, this dataset has more character classes, more flexible styles, longer text and less training images.

Analysis of Character Synthesis

Network architecture. [Brock et al., 2019] reports GANs benefit from wider network and larger batch size. We also conduct similar experiments (Table 3) on character model. To test the impact of network parameters, we gradually increase the number of channels (rows 1 - 5). However, we do not notice wider networks obviously improve the performance, even the largest model (row 5) is slightly worse than a small one (row 4). Further, the batch size is doubled from 128 to 2048 (row 4, 6-9) 1 . Also, this trick does not improve the performance simultaneously in IS and FID. We conjecture that the scaling trick explored in ImageNet suffers from saturation due to the limited training data in IC03+IIIT dataset. Therefore, we adopt the configuration in row 4 considering the trade-off between image quality and training efficiency.

1 We implement batch size 1024, 2048 by accumulating gradients, thus the statistics of BatchNorm is computed at batch size 512.

Style loss. Further, we investigate the effectiveness of proposed style loss by adjusting λs in Equation 6,7. As shown in Figure 4, λs is set to 0, 0.5, 1, 2, respectively. Note that λs = 0 equals to training Gc without style loss. By comparing Figure 4a with 4c,4d, we can obviously observe that the style loss essentially unifies the font, color, background, etc. of the characters. However, a small λs (Figure 4b) is too weak to control the style. Though a big λs (Figure 4d) achieves satisfactory style appearance, we find it slowing down the convergence slightly. Thus, λs = 1 is adopted in our further experiments. Besides, the style loss does not harm performance according the IS and FID scores.

Analysis of Word Synthesis

Ablation study. Applying vanilla GAN with U-Net as the generator and removing the latent vector, we build a simple baseline. Based on this baseline, we illustrate the effectiveness of each component of the proposed method. Table 4 summarizes the comparison results, where L1 is the loss defined in Equation 8 (λl = 1); patch represents the patch discriminator; pretrained indicates initializing Dw using Dc; Dw(·, y) and Gw(·, y) are imposing condition y on discriminator and generator respectively. As can be seen from Table 4, without L1 and patch (rows 1,2), the models fail to generate plausible images. The configuration of 3rd row is quite similar to [Shrivastava et al., 2017] as L1 and patch are the principal components of [Shrivastava et al., 2017]. Further, we gain benefit from using pretrained Dw (4th row). Lastly, injecting y to Dw substantially improves the IS and FID scores, and Gw conditioned on y also boosts the IS score.

L1 loss. Based on the results of ablation study, we further analyze the impact of L1 self-regularization, which is important to retain character appearance from the character model. As reported in Table 5, a big λl typically reduces the image diversity and a small one is hard to keep the character skeleton. Therefore, we choose λl = 5 as it performs the best performance both in realism and variation.

Scene text recognition. Finally, we apply the proposed STS-GAN to scene text recognition [Fang et al., 2018; Xie et al., 2019]. To obtain high-quality images, we only keep the synthetic images which are classified correctly by the word classifier with each character confidence greater than 0.9. Therefore, we build a synthetic dataset, used for training the word classifier together with IC03+IIIT dataset. From the accuracy comparisons in Figure 5 we can see, training with additional synthetic images can boost the performance for word classifiers (up to 3.1% for 10k), which also demonstrates the verisimilitude of synthetic images as unreadable images will result in turbulent training. However, more synthetic images do not help to improve accuracy linearly. We speculate training with limited data, the image diversity of our method is unable to support the accuracy growth in large scale, which remains further study. In total, starting from only 3096 training word images, our STS-GAN is able to synthesize readable images and boost the recognition performance for a mainstream algorithm.

4.4 Comparison with Other Models

Finally, we compare the performance of the proposed model with non-learning based methods and traditional GAN mod-

(b) Character model + Pix2Pix

Figure 7: Samples from traditional GAN models.

els, and the IS and FID scores are recorded in Table 6. The non-learning methods [Jaderberg et al., 2014; Gupta et al., 2016; Zhan et al., 2018] achieve impressive IS and FID scores as they are effective to generate readable text images. Note that our method synthesizes text images from scratch without complicated rendering process. Compared to the non-learning methods, our STS-GAN obtains better IS and FID scores, which denotes that STS-GAN not only generates high-quality images, but also captures an approximately correct distribution as the desired image distribution (i.e., IC03+IIIT dataset in this case).

In addition, two learning-based methods are prepared to adapt our task. The first is GAN-CLS [Reed et al., 2016], which is a conditional GAN designed for description text to image synthesis. Sentence embedding in GAN-CLS is replaced with the embedding of text label y to make it support our task, and y is embedded using ELMo [Peters et al., 2018]. The second is a two-stage method using Pix2Pix [Isola et al., 2017] to replace our word synthesis. Figure 7 displays some samples and Table 6 gives corresponding IS, FID scores for the above two models. The GAN-CLS (Figure 7a) merely generates confusing and meaningless images, as vanilla GANs are hard to recover character structure. Also, for the popular image-to-image framework (Figure 7b), it is difficult to retain clear text appearance obtained from the character model. Compared to these models, our welldesigned STS-GAN can successfully synthesize images with clear and readable text.

4.5 Discussion

Drawing text in natural images is a challenging problem. Though STS-GAN is demonstrated to be effective, we still observe some failure cases in IC03+IIIT dataset, which rarely occur in easier SVHN dataset. Generally, with a stitched image ˆxw in which text is legible comes a well-drawn xw. In some cases, we have noticed successful images converted from poor ˆxw (Figure 6). Nevertheless, failure conversion will cause a clear ˆxw to an unsatisfactory xw, and a blurring ˆxw typically leads to failure synthesis (Figure 8). Besides, we note that in character synthesis stage, due to the limited training samples, classes with low character frequency ('Q', 'q', 'j', etc.) encounter mode collapse, which may also harm the final performance.

5 Conclusion

In this paper, we propose scene text synthesis GAN (STSGAN) for generating text images. To cope with the problem that vanilla GANs are difficult to model structural patterns, we first generate local character structure which is constrained by style loss to unify text style. Then local characters are converted into plausible word image through pixel-wise manipulation. We conduct experiments on SVHN and ICDAR, IIIT5K datasets, showing STS-GAN can synthesize visually pleasing text images. In the future, we will extend this method to full scene text images rather than cropped images.

Acknowledgments

This work is supported by National Key R&D Program 2016 (Grant No. 2016YFB0801305), National Defense Science and Technology Fund for Distinguished Young Scholars (2017-JCJQ-ZQ-022), the National Nature Science Foundation of China (61525206, 61771468), the Youth Innovation Promotion Association Chinese Academy of Sciences (2017209).