DSRN: A Deep Scale Relationship Network for Scene Text Detection

1 Introduction

Recently, extracting and recognizing textual information in the wild have become increasingly important and popular because of its significant value in practical applications [Long et al., 2018; Lyu et al., 2018; Fang et al., 2018; Xie et al., 2019]. Scene text detection, playing a critical role in the whole process, is one of the main bottlenecks in recognition quality.

* Hongtao Xie (htxie@ustc.edu.cn)

Figure 1: We visualize the feature maps of different channels in training stage. The proposed method can obtain more uniform activation of multi-size text instances. Both models use ResNet50 as their basic network.

Despite the fact that recent multi-scale object detection algorithms [Liu et al., 2016; Zhou et al., 2018] gain great improvements, directly implementing these approaches for scene text detection, which are only for horizontal targets, may not be a good choice for multi-oriented text detection.

In the past few years, detecting objects with large-variance scale has been the main challenge in both general object detection and scene text detection. To address this problem, SSD [Liu et al., 2016] combines predictions from multi-scale features to handle the objects with various scales. SAN [Kim et al., 2018] constructs scale normalized patches to reduce the scale variation of objects in feature maps. Besides the scale variance, multi-oriented detection is another challenge in scene text detection. TextSnake [Long et al., 2018] uses the sequence of overlapping disks to represent text instances of arbitrary orientation. Lyu et al. [2018] propose a novel approach for multi-oriented scene text detection by predicting the corners of a text instance. However, these previous methods usually simply fuse multi-scale features and require fairly complicated post processing, which would limit the performance and result in large amount of time consumption.

In this paper, we propose an end-to-end architecture called Deep Scale Relationship Network (DSRN) to handle the large-variance scale problem in scene text detection by mapping multi-scale convolution features onto a scale invariant space, which obtains uniform activation of multi-size text instances. Although different-scale features facilitate the detection of multi-size objects, simply concatenating features with multi-scale information would limit the performance of network due to the heterogeneity of features. To handle this issue, we firstly develop a Scale-transfer module to transfer multi-scale features to a unified dimension. Then a Scale Relationship module is constructed on different layers to pass contextual scale information through bidirectional convolution operations. In the end, the proposed method essentially improves the quality of convolutional features by obtaining uniform activation of multi-size text instances (Figure 1 (c)).

The contributions of this paper are following:

• We develop a Scale Relationship module for multi-scale feature aggregation, which maps multi-scale convolution features onto a scale invariant space and obtains uniform activation of multi-size text instances.
• We propose a new loss function called Recall Loss, which up-weights the loss assigned to poor-classified examples. It efficiently reduces the miss-detected instances and boosts performance in recall.
• Our DSRN essentially improves the quality of convolutional features, and the proposed Scale Relation module can be easily embedded into other existing detection networks, boosting the performance without obvious speed sacrifice.

2 RelatedWorks

2.1 Indirect Regression based methods.

Indirect Regression based methods regress the offsets from a default box to the corresponding ground truth. As inspired by the object detection method [Liu et al., 2016], Liao et.al.[2018] construct a fully convolutional architecture and adopt ARF [Zhou et al., 2017b] to generate rotationsensitive features, which obtains better representation for multi-oriented texts. He et.al.[2017a] develop a hierarchical inception module to aggregate multi-scale features, then predictions from multi-scale features are combined for better multi-size detection.

2.2 Direct Regression based methods.

In order to handle the large variance of text scale and multi-orientation issues, many previous works often contain multiple sequential steps or complicated post processing, which result in large time consumption and are difficult for practical application. Different from previous methods, direct regression based methods directly predict offsets from bounding box boundaries or vertexes to points without setting complex hand-crafted hyperparameters (e.g. scale of default boxes), providing a new approach for accurate and efficient text detection [Zhou et al., 2017a; He et al., 2017b].

Although direct regression based methods provide an efficient approach for scene text detection, simply concatenating features from different levels would limit the network performance due to the heterogeneity of features. Thus we propose an effective approach to aggregate multi-scale information through bidirectional convolution operations, which maps multi-scale convolution features onto a scale invariant space. Contributing to the aggregation of multi-scale information, our DSRN obtains more uniform activation of multi-size text instances (Figure 1).

2.3 Segmentation based methods.

Inspired by the segmentation methods [Long et al., 2015; Min and Chen, 2018], some algorithms are proposed for scene text detection by using segmentation maps. TextSnake [Long et al., 2018] uses the sequence of overlapping disks to represent text instances of arbitrary orientation. Pixel link [Deng et al., 2018] predicts the links among every pixel and their neighbors which are valid when both of the linked pixels belong to text instances. By doing this, pixel link successfully separates text instances that are very close to each other. Although the segmentation based methods are able to handle the text with various scales, they usually require fairly complicated post processing which will slow down the speed.

3 The ProposedMethod

3.1 Overview

In this section, we will introduce the details of DSRN. Our method is based on [Zeng et al., 2016], which is originally proposed to use the contextual visual cues of ROIs in general object detection. We design a novel Scale Relationship module and implement a Scale-transfer module to extend this framework for scene text detection. Different from [Zeng et al., 2016], our method is proposed to aggregate multi-scale feature maps and to map the multi-scale features onto a scale invariant space to obtain uniform activation of multi-size text instances. The total architecture of DSRN is illustrated in Figure 2.

We use ResNet50 [He et al., 2016] as our basic network and reduce the channels of feature maps during up-sample to obtain a light framework. In order to aggregate multiscale information from feature maps of different scales, Scale-transfer module is proposed to transfer the multi-scale convolution features to a unified dimension. Then Scale Relationship module passes contextual multi-scale information between unified features during both feature learning and extraction. Such message passing is conducted in different layers and carried out through bidirectional convolution operations. In the end, the Non-maximum suppression (NMS) is adopted to reduce redundant results and the whole network is optimized in an end-to-end way. The effective aggregation of multi-scale information and lightness of this architecture make our approach accurate and efficient.

3.2 Feature Extraction

The backbone of DSRN is adapted from a pre-trained ResNet50 [He et al., 2016] network and designed with following considerations: 1) the backbone must have enough capacity to handle the large variance of text scale. 2) Features should contain more contextual information, due to the complexity of background within natural scene images. 3) The corresponding feature maps should include strong semantic information to represent multi-size texts. Inspired by FPN [Lin et al., 2017a] which achieves the good performance on those problems, we adopt the backbone with a similar architecture to extract features.

Particularly, we convert the fc layers in ResNet50 to an attention module [Wang et al., 2017] (F1 in Figure 2). Then several extra convolutional layers (F2, F3 and F4) are stacked above the attention module in a top-down pathway.

3.3 Scale-transfer Module

Features from different stages have various scales, which makes it difficult for aggregation. Inspired by the object detection method [Zhou et al., 2018], we develop a Scaletransfer module to generate feature maps with a unified dimension (1/4 of input image in our experiment).

As illustrated in Figure 3, channel matching layer firstly produces features with corresponding channels, which controls the output channels of scale-transfer layer. Then scale-transfer layer expands width and hight of feature maps simultaneously by compressing the number of channels. The dimensions of input tensor are Ci×Hi×Wi (i = 1, 2, 3) and those of outputs are Ci m2 ×m×Hi ×m×Wi. We choose m to 8, 4, 2 for F1, F2, F3 respectively. Finally, the multi-scale features are mapped to a unified dimension through Scaletransfer module.

As channel matching layer should also be implemented following other expanding approaches for channel normalization (channels of features are usually related to stages), Scale-transfer module expands the scale of feature maps with less additional parameters, which can make our network more efficient.

3.4 Scale RelationshipModule

The detection of large text requires features from late-stage with small scales, while accurate geometry prediction of small texts needs low-level information from early-stage [Zhou et al., 2017a]. However, simply concatenating features with multi-scale information would limit the performance of network due to the heterogeneity of features. Different from previous methods [He et al., 2017a; Lyu et al., 2018; Liao et al., 2018] which combine predictions from multiscale features to handle the large-variance scale problem, we propose a Scale Relationship module to aggregate multi-scale features. It passes contextual scale information among multiscale feature maps and maps the multi-scale features onto a scale invariant space, which results in uniform activation of multi-size text instances and essentially improves the quality of features (Figure 1(c)).

As illustrated in Figure 4, Scale Relationship module passes contextual scale information by sequentially convoluting feature maps through bidirectional convolution operations. f0, f1, f2 and f3 are corresponding features from decoding layers F1, F2, F3 and F4 in Figure 2 respectively. Convolution in the first direction starts from the last decoding layer (f3) and ends at the first decoding layer (f0), thus the sequential feature maps can receive scale information from larger-scale features, which is better for small text detection. In contrast, sequential features can also receive large-size textual information in the second direction. Element-wise product operation is learned to control how much contextual scale information can be received from previous feature maps. The outputs of bidirectional convolution (gi and hi, i = 0, 1, 2, 3 in Figure 4) have richer multi-scale and stronger semantic information than previous feature maps (fi, i = 0, 1, 2, 3).

In the end, we concatenate the output features from bidirectional convolution operations and implement two sequential convolution layers to obtain better representation. In order to balance the accuracy and efficiency in the detection task, only four decoding layers are used in the proposed method.

4 Loss Functions

The multi-task loss is formulated as:

Where Lcls and Lreg are classification loss and regression loss respectively. λreg is a hyperparameter to balance Lcls and Lreg. In our experiment, we set λreg to 1.

4.1 Loss for Clssification

In most previous detection approaches, processes such as hard negative mining and balanced sampling are implemented to handle the imbalanced distribution between background and target objects. Lin et al. [2017b] propose a Focal Loss (FL), which achieves better training by down-weighting the well-classified examples, as seen in (2).

As direct regression based methods directly predict a score map of text regions, Focal Loss will distribute almost the same weight to miss-detected text instances (red region in Figure 6) and the border of text instances (green region in Figure 6), which makes the model pay identical attention to both regions during training and is harmful to the performance. To handle this issue, we propose a new loss function to force the network to pay more attention to the miss-detected text instances without concerning about the border of texts, which is called Recall Loss (RL).

As seen in (3)(4)(5). Where S is the predicted region with confidence of pixels above 0.5, G is the corresponding ground truth, e is a constant, and α and β are hyperparameters which will be illustrated in ablation studies. If IoU < β , then α > 1 exists, which means all the pixels in this region will be assigned a larger weight, and the border and the center of text instances with IoU > β will obtain the same weight that is lower than miss-detected text instances. This property forces the network to focus on regions with small IoU, which can effectively reduce the miss-detected instances (Figure 7 (a)). η1 and η2 are balancing factors between positive and negative samples. In our experiment, we combine Recall Loss with Dice Loss (6) and balance them by λR and λD respectively (oi and yi are pixels in score map and ground truth respectively). In the end, the classification loss is formulated as (7).

4.2 Loss for Regression

We choose the same regression approach in [Zhou et al., 2017a], which is invariant against scales of objects (8)(9).

We combine Lloc with Lθ for final regression loss. Distance from a pixel to 4 boundaries and value of angle are outputs for both training and inference.

5 Experiment

In this section, we evaluate the performance of Scale Relationship module and of Recall Loss respectively. We compare our DSRN with recent state-of-art methods on benchmark datasets to prove our accuracy and efficiency.

5.1 Benchmark Datasets

ICDAR2015. This is a dataset for incidental scene text detection proposed in the Challenge 4 of ICDAR 2015 Robust Reading Competition [Karatzas et al., 2015]. It includes 1000 training images and 500 test images with annotations labeled as 4 vertices of a word level quadrangle. We fit a rotated rectangle with the minimum area for training. Different from previous dataset with only horizontal annotations, this benchmark is proposed for evaluating multi-orientation text detection and contains text of different scales, ambiguities, resolutions, perspectives, and directions.

HUST. [Yao et al., 2014] is a dataset contains 400 images, which consists of Arabic numbers and English letters of different fonts with text line level labels.

MSRA-TD500. This is a dataset proposed in [Yao et al., 2012] for detecting arbitrary-oriented and multi-lingual long text lines. It contains 300 images for training and 200 images for testing. Since the size of training data is too small to learn a deep newtwork, we also use 400 images from HUST [Yao et al., 2014] in training stage.

5.2 Experimental Setup

Our proposed network is trained end-to-end on NVIDIA TITAN X GPU using ADAM[Kingma and Ba, 2014] optimizer. We perform data augmentation by randomly cropping each image and resize it to 512 × 512 for training. We update the learning rate by a multi-step strategy. The initial learning rate is 1e-3, and decays by 0.94 every 10k steps. We set batchsize to be 14 and training continues until convergence. In test stage, NMS is implemented to reduce the redundant results.

5.3 Ablation Studies

Recall Loss. θ and β are important hyperparameters which allow us to vary the capacity of Recall Loss in our model. To investigate this relationship, several experiments are conducted using different θ and β values on ICDAR2015 in Table 1. We also conduct an experiment without Recall Loss (α = 1 in (3)) for optimizing. The comparison reveals that Recall Loss can boost the performance in recall and obtain higher F-measure. Furthermore, we compare Recall Loss with Focal Loss [Lin et al., 2017b] and Lovasz-Softmax Loss [Berman et al., 2018] in Table 2. The comparison reveals that Recall Loss achieves better performance in both recall and F-measure.

Scale Relationship module. To explore the gain of our Scale Relationship module, we train baseline networks without Scale Relationship module (baseline1) and with unidirectional convolution operation (baseline2). All baseline models contain the identical backbone, prediction module and training settings as DSRN. With slight time consumption, DSRN boosts the performance greatly.

5.4 Comparison with existing methods

We evaluate our model on ICDAR2015 incidental scene text to test its performance of arbitrarily oriented text detection. 229 images from ICDAR2013 are also used for training. We choose θ and β to 1.7 and 0.3 respectively in training stage and resize images to 768 × 1280 in test stage .

We compare the proposed method with state-of-art algorithms in Table 4. DSRN works better in direct regression based methods [He et al., 2017b; Zhou et al., 2017a], and outperforms them by a large margin (0.814 vs 0.70 and 0.782). We attribute our high performance to the aggregation of multi-scale information in Scale Relationship module, which essentially improves the quality of feature maps. When tested at single scale, the proposed method surpasses most state-of-art methods [Lyu et al., 2018; Wang et al., 2018; Liu et al., 2018] with the fastest speed. Although [Liao et al., 2018] is slightly better than ours (0.822 vs 0.814), which contains a corner grouping process resulting a lot of time consumption, our method achieves 35% faster speed (8.8 FPS vs 6.5 FPS) due to a lighter architecture with simpler post processing (NMS).

On MSRA-TD500, we evaluate the capacity of our algorithm for detecting long and multi-lingual text lines. We pre-train our model on ICDAR2015 and then finetune it until convergence. In test stage, we resize images to 672×672.

With comparisons to other representative results in Table 5, our method achieves the state-of-art performance in recall, precision and F-measure (0.712, 0.876 and 0.785), outperforming the direct regression based methods [He et al., 2017b; Zhou et al., 2017a] (0.785 vs 0.74 and 0.761) and other representative methods [Long et al., 2018; Deng et al., 2018]. Furthermore, our method achieves the identical stateof-art performance in recall to [Lyu et al., 2018] with 142% faster speed (13.3 FPS vs 5.7 FPS).

Particularly, [Lyu et al., 2018] and [Liao et al., 2018] use SynthText [Gupta et al., 2016] (800000 images in [Lyu et al., 2018]) to pre-train their models and then finetune them on corresponding tasks (both ICDAR2015 and MSRA-TD500). The proposed DSRN can obtain better or competitive results with much less training datasets, which proves that our DSRN is a more general model for scene text detection task.

Some detection samples of DSRN are visualized in Figure 7. The proposed method can perform well in most situations, handling the large variance of text scale and arbitrary orientation in scene text detection. However, it fails to detect text lines with large character spacing (top of Figure 7 (d)). Furthermore, our method performs not well in curved text detection (bottom of Figure 7(d)), as few curved samples are in training set.

5.5 Rationality of High Performance and Fast Speed

DSRN is proposed to detect multi-size texts automatically. The huge increase in accuracy and efficiency is mainly due to three aspects. 1) We develop a Scale Relationship module to learn the contextual scale information and to obtain uniform activation of multi-size text instances, which essentially improves the quality of convolutional feature maps. 2) We propose a new loss function called Recall Loss, which effectively reduces miss-detected text instances by upweighting the weight of poor-classified examples. 3) The direct regression approach and light network make our model efficient.

6 Conclusion

In this paper, we have presented a novel end-to-end method for scene text detection. The main idea is mapping multiscale features onto a scale invariant space, which obtains uniform activation of multi-size text instances and effectively handles the problem of large variance of text scale in scene text detection without setting complex hand-crafted hyperparameters. Another improvement is that we propose a new loss function called Recall Loss, which reduces missdetected text instances by up-weighting the poor-classified examples. The experiments on benchmarks reveal that our model achieves the state-of-art performance with impressive speed. Furthermore, we also analyze the reasons of our high performance and fast speed. As for future work, we will further improve the performance by combining our detection framework with a recognition branch.

Acknowledgments

This work is supported by the National Key Research and Development Program of China (2017YFC0820600), National Defense Science and Technology Fund for Distinguished Young Scholars (2017-JCJQ-ZQ-022), the National Nature Science Foundation of China (61525206, 61771468), the Youth Innovation Promotion Association Chinese Academy of Sciences (2017209).