TextFuseNet: Scene Text Detection with Richer Fused Features

1 Introduction

Scene text detection is attracting increasing attention in the computer vision community. Many progress has been made with the rapid development of deep learning[Wang et al., 2019d][Wang et al., 2017][Qiao et al., 2019][Gao et al., 2019]. However, this task remains challenging, since the texts commonly have diversified shapes and text detectors can be easily affected by the issues like complex backgrounds, irregular shapes, and texture interference.

∗Corresponding author

Existing methods mainly have two types: character-based methods and word-based methods. Character-based methods regard the text as a combination of multiple characters. They first extract characters with well-designed character detectors and then group them into words. However, characterbased methods are generally time-consuming due to the significantly large number of character candidates generated for text detection. Instead of character-based methods, wordbased methods have been proposed to directly detect words based on generic object detection pipelines. Although they are much simpler and more efficient, these methods usually fail to effectively detect the texts with arbitrary shapes. To tackle this issue, some word-based methods further apply instance segmentation to conduct text detection. In these methods, foreground segmentation masks are estimated to help determine various text shapes. Despite promising results, existing instance-segmentation-based methods still have two major limitations. Firstly, these methods only detect texts based on a single region of interest (RoI) without considering global contexts, thus they tend to produce inaccurate detection results based on limited visual information. Secondly, prevailing methods do not model different levels of word semantics, running the risk of producing false positives for text detection. Figure 1 shows an example of these methods.

In this paper, we propose a novel scene text detection framework, namely TextFuseNet, to effectively detect texts with arbitrary shapes by utilizing richer fused features. In general, we follow Mask R-CNN [He et al., 2017] and Mask TextSpotter [Lyu et al., 2018] and formulate the text detection task as an instance segmentation task. Unlike these methods, we recast the original pipeline of Mask R-CNN to enable the analysis and fusion of three levels of feature representations, i.e., character-, word-, and global-level features, for text detection. In particular, we first introduce an additional semantic segmentation branch to the detection pipeline to help perceive and extract global-level representations. The global semantic features can later be used to guide the detection and mask branches of the detection pipeline. Next, we attempt to extract character- and word-level features within detection and mask branches in the Mask R-CNN pipeline. Different from the original Mask R-CNN, in the detection and mask branches, we detect and segment not only word instances but also character instances, delivering both character- and wordlevel representations. After the perception of three-level representations, we then introduce the multi-path feature fusion architecture, which fuses the character-, word- and globallevel features through a multi-path fusion network, to facilitate the TextFuseNet to learn a more discriminative representation and produce more accurate text detection results. In practice, considering that some existing datasets lack annotations for characters, we further develop a weakly supervised learning scheme to generate character-level annotations by learning from word-level annotated datasets. Overall, the architecture of TextFuseNet is shown in Figure 2.

The contributions of this work are three-fold: (1) We propose a novel framework, namely TextFuseNet, which extracts character-, word- and global-level features and introduce a multi-path fusion architecture to fuse them for accurate text detection; (2) Based on the proposed framework, we introduce a weakly supervised learning scheme, which exploits word-level annotations to guide searching for character training samples, to achieve effective learning without annotations for character instances; (3) Our proposed framework achieves state-of-the-art performance on several famous benchmarks which contain texts of arbitrary shapes.

2 RelatedWork

As mentioned above, existing methods can be roughly classified into two main categories, i.e., character-based methods and word-based methods.

Character-based methods usually first apply some complicated character detectors, such as SWT, MSER and FASText, to extract character candidates. These character candidates are filtered by a character/non-character classifier to remove false candidates. Finally, the remained characters are grouped into words according to either prior knowledge or some clustering/grouping models. However, most characterbased methods require elaborate design and involve multiple stages of processing, which are very complicated and lead to error accumulation. Therefore, the performance of characterbased methods is always time-consuming and suboptimal.

Word-based methods detect words directly, which are mainly inspired by general object detection methods. [Tian et al., 2016] proposed a Connectionist Text Proposal Network (CTPN) which consists of CNN and RNN to detect whole text lines by linking a series of small text boxes. Inspired by SSD, [Liao et al., 2018a] proposed TextBoxes and its extension TextBoxes++ by adding several text-box layers. [Shi et al., 2017] proposed SegLink by employing Fully Convolutional Networks (FCN) to detect text segments and their link relationships. Text segments are linked as the final detection result according to their relationship. However, these methods are only suitable for horizontal or multi-oriented text.

In order to tackle the challenge of texts with arbitrary shapes, many instance-segmentation-based methods have been proposed to detect text with arbitrary shapes. [Deng et al., 2018] conducts text/non-text prediction and link prediction through CNN, and connects positive sample pixels with positive links, which directly obtains text boxes without regression. [Xie et al., 2019] proposed a Supervised Pyramid Context Network (SPCNet) to locate text regions based on Mask R-CNN. [Wang et al., 2019a] proposed a Progressive Scale Expansion Network (PSENet) to detect text with arbitrary shapes. [Tian et al., 2019] mapped pixels onto an embedding space and introduced a shape-aware loss to make training adaptively accommodate various aspect ratios of text instances. Compared with previous works, we analyze and fuse more different levels of features to obtain richer fused features, which effectively improve the performance of text detection.

3 Methodology

In this section, we describe how the multi-level feature representations are extracted by semantic segmentation, detection, and mask branches, and how we fuse them using the multi-path fusion architecture. Meanwhile, we also explore the strategy of weakly supervised learning for generating character-level annotations.

3.1 Framework

Figure 2 depicts the overall architecture of TextFuseNet. In the TextFuseNet, we first extract multi-level feature representations and then perform multi-path fusion to conduct text detection. This framework is mainly implemented by five components: a feature pyramid network (FPN) as backbone for extracting multi-scale feature maps, a region proposal network (RPN) for generating text proposals, a semantic segmentation branch for exploiting global semantics, a detection branch for detecting words and characters, and a mask branch for instance segmentation of words and characters.

In TextFuseNet, we first follow Mask R-CNN and Mask TextSpotter and employ a ResNet as the backbone of FPN. Also, we use RPN to generate text proposals for the subsequent detection and mask branches. Then, to extract multilevel feature representations, we mainly propose to apply the following implementations. First, we introduce a new semantic segmentation branch to conduct semantic segmentation for the input image and help obtain global-level features. Then, in the detection branch that refines text proposals via predicting their categories and adopting bounding box regression, we extract and fuse word- and global-level features to detect both words and characters. This is different from existing methods only focus on detecting a single word or character for each proposal. For the mask branch that performs instance segmentation for objects detected from the detection branch, we extract and fuse all the character-, word-, and global-level features to fulfill the instance segmentation, as well as the final text detection task. The detailed network configurations to extract multi-level feature representations are presented in Section 3.2. After the multi-features are extracted, we then propose a multi-path fusion architecture to fuse the different features for detecting texts with arbitrary shapes. The multipath fusion architecture can effectively align and merge the multi-level features to deliver robust text detection. The details of the implementation of the multi-path fusion architecture are described in Section 3.3.

3.2 Multi-level Feature Representation

In general, character- and word-level features can be easily obtained within the detection and mask branches of the detector. We can achieve this by detecting both words and characters appeared within the proposals. RoIAlign is applied here to extract different features and perform detection for both words and characters.

However, we need a novel network in the feature extraction stage to help obtain the global-level features. Therefore, we propose to further employ a semantic segmentation branch in the detector to extract global-level features. As presented in Figure 2, the semantic segmentation branch is constructed based on the output of FPN. We fuse features from all the levels into a unified representation and perform segmentation on this unified representation, thus obtaining globally segmented results for text detection. In practice, we apply a 1×1 convolution to align channel numbers of features from different levels and resize the feature maps into the same size for later unification.

3.3 Multi-path Fusion Architecture

After we have obtained multi-level features, we adopt multipath fusion in both detection and mask branches. In the detection branch, based on the text proposals obtained from RPN, we extract global- and word-level features for text detection in different paths. We then fuse both types of features to deliver text detection in the form of words and characters. Note that we cannot extract and fuse character-level features in detection branch because the characters are not yet recognized before performing the detection. In practice, given a generated text proposal, we use RoIAlign to extract the global- and word-level features within the size of 7×7 from the output features of FPN. We fuse these features via by element-wise summation, and feed them into a 3×3 convolution layer and a 1×1 layer. The final fused features are used for classification and bounding box regression.

In the mask branch, for each word-level instance, we can then fuse the corresponding character-, word-, and globallevel features within the multi-path fusion architecture for instance segmentation. Figure 3 shows a detailed illustration of the multi-path fusion architecture. In the proposed architecture, we extract multi-level features from different paths and fuse them to obtain richer features to help learn a more discriminative representation.

Formally, given an input word denoted by ri, we first identify the character results Ci which belong to this word proposal based on its ratio of the intersection with a character over the character's area, meaning that the ratio is 1 if the word box fully covers the character, and 0 if not. We use cj to denote the characters. Then the set of characters Ci belong to the word ri can be collected based on:

where bi and bj are the bounding boxes of word ri and character instance cj, respectively, and T is the threshold. In our implementation, we set T = 0.8.

Since the number of characters is not fixed and may range from zero to hundreds, for a given detected word ri, we fuse the features of characters in the set Ci into a unified representation. In particular, we first use RoIAlign to extract the corresponding features with the size of 14×14 for each character in Ci and then fuse these feature maps via element-wise summation. Through a 3×3 convolution layer and a 1×1 convolution layer, we can obtain the final character-level features.

By further applying RoIAlign to extract features of words and corresponding global semantic features, we fuse all these three levels features via element-wise summation, and feed them into a 3×3 convolution layer and a 1×1 layer to obtain richer features. The final fused features are used for instance segmentation. Note that the 3×3 convolution layers and 1×1 convolution layer following element-wise summation are used for further bridging the semantic gap among different features.

Overall Objective. Lastly, we formulate the overall objective of the proposed TextFuseNet for tackling text detection problem:

where Lrpn, Lseg, Ldet and Lmask are the loss function of RPN, semantic segmentation branch, detection branch and mask branch respectively.

3.4 Weakly Supervised Learning

Since TextFuseNet is formulated to detect both words and characters, character-level annotations are needed to achieve effective training. However, as mentioned previously, some existing datasets do not provide character-level annotations to train the TextFuseNet. Instead of annotating characters which is a time-consuming and labor costly task, we are inspired by the idea of weakly supervised learning and propose a weak supervision-based learning scheme to help train the TextFuseNet. In the proposed scheme, we search for characterlevel training examples by learning from the weakly supervised data with a pre-trained model. The pre-trained model is trained based on our proposed framework on a fully annotated datasets which provide both character- and word-level annotations. Then, for a dataset A that only has word-level annotations, the goal of our developed weak supervised learning is to search for character training samples in A through the pre-trained model M.

More specifically, we first apply the pre-trained model M on the word-level annotated dataset A. For each image in dataset A, we can obtain a set of character candidate samples:

where ci, si, bi, andmi represent the predicted category, confidence score, bounding box, and mask of the i-th character candidate sample ri, respectively. Then we filter false positive samples in R based on the confidence score threshold and the weakly supervised word-level annotations and obtain positive character samples:

where C denotes all character categories to be detected, S represents the confidence score threshold used to identify positive character samples, (mi∩gi) mi denotes the intersection overlap of candidate character sample ri with its word-level ground truth gj , and T is the threshold to determines whether the candidate character sample is inside the words or not. Due to the constraints provided by word-level annotations, the confidence score threshold S can be set to a relatively lower, which is also beneficial for keeping the diversity of character samples. In our implementation, S and T is set to 0.1 and 0.8 respectively. Finally, the identified positive character samples can be used as character-level annotations and be combined with the word-level annotations to train a more robust and accurate text detection model.

4 Experiments

In this section, we evaluate the performance of TextFuseNet on four challenging public benchmark datasets: ICDAR 2013, ICDAR 2015, Total-Text and CTW-1500, and compare with previous state-of-the-art methods.

4.1 Datasets

SynthText is a synthetically generated dataset and usually used for pre-training the text detection models. This dataset consists of 800,000 images with 8 million synthetic word instances with both word- and character-level annotations in the form of rotated rectangles.

ICDAR2013 is a typical horizontal text dataset and is proposed in Challenge 2 of the ICDAR 2013 Robust Reading Competition. It contains 229 training images and 233 test images. ICDAR 2013 also provides both character- and wordlevel annotations.

ICDAR2015 is a multi-orient text dataset and is proposed in Challenge 4 of the ICDAR 2015 Robust Reading Competition. It focuses on incidental scene text and contains 1000 training images and 500 test images. This dataset only provides word-level annotations labeled with quadrangles.

Total-Text is a comprehensive arbitrary shape text dataset for scene text reading. Total-Text contains 1255 training images and 300 test images. All images are annotated with polygons in word-level.

CTW-1500 also focuses on arbitrary shape text reading and contains 1000 training images and 500 test images. Different from Total-Text, the annotations in CTW-1500 are labeled with polygons in the text-line-level.

4.2 Implementation Details

We implemented our framework based on the Maskrcnnbenchmark, and all experiments are conducted on a highperformance server with NVidia Tesla V100 (16G) GPUs.

The model is trained with 4 GPUs and evaluated with 1 GPU.

Training. The whole training process was divided into three stages: pre-training on SynthText, searching for character training samples under weak supervision, and fine-tuning on the real-world data. Since SynthText provides both wordand character-level annotations, we can obtain a pre-trained model with full supervision. After pre-training, for weakly supervised learning, we apply the pre-trained model on ICDAR 2015, Total-Text and CTW-1500 to search for character training samples of their corresponding word-level annotations. The identified character samples are then combined with their original word-level annotations to fine-tune the pretrained models on the new datasets. To better analyze the capability of the proposed TextFuseNet, we adopt ResNet with two different depths of {50, 101} as the backbone on each dataset. Moreover, in order to enhance network robustness, data augmentation strategies such as multi-scale training, rotating randomly, and random color adjusting are applied.

Stochastic gradient descent (SGD) is adopted to optimize our framework. The weight decay is set to 0.0001, momentum is set to 0.9, and batch size is set to 8. In the pre-training stage, we train the model on SynthText for 20 epochs. The learning rate is set to 0.01 in the first 10 epochs, divided by 10 in the last 10 epochs. In the fine-tuning stage, the training iterations on every dataset are set to 20K. The learning rate is set to 0.005 in the first 10K iterations, divided by 10 in the remains.

Inference. During inference, the shorter side of the test image was scaled to 1000 while keeping the aspect ratio unchanged. The global semantic features are extracted in the semantic segmentation branch. For RPN generated text proposals, we select the top 1,000 proposals for the detection branch. With the detection results obtained, we adopt Soft NMS to suppress the redundant bounding boxes. The instance segmentation is then performed upon suppressed detection results. We only keep the instance segmentation results of word instances as the final text detection results.

4.3 Ablation Study

Compared with the original Mask R-CNN, we introduce two modules to improve the performance of text detection in our proposed TextFuseNet. The first module is to conduct multilevel feature representation (MFR). The other is introducing multi-path features fusion architecture (MFA) to obtain richer fused features for text detection. Therefore, we conducted an ablation study on ICDAR 2015 and Total-Text to evaluate how each module in TextFuseNet influences final performance. For each dataset of ICDAR 2015 and Total-Text, three models are trained and the comparison results with different models are shown in Table 1. "Baseline" refers to the model trained with the original Mark R-CNN. "MFR" represents the model trained with Mask R-CNN using multi-level feature representation, and "MFR+MFA" refers to the model with full implementation of TextFuseNet. The backbone network used in this ablation study is a FPN with ResNet-50.

As is shown in Table 1, multi-level feature representation alone improves both precision and recall significantly, and the final improvements of "MFR" are more than F-measure of 2% on both ICDAR 2015 and Total-Text. Moreover, the combination of "MFR" and "MFA" can further enhance the performance and improves the F-measure by 4.6% and 4.3% beyond the baseline on ICDAR 2015 and Total-Text respectively. These results validate that both multi-level feature representation and multi-path features fusion can help to obtain richer fused features and more discriminative representations which are beneficial for text detection.

4.4 Comparisons with State-of-the-Art Methods

Arbitrary Shape Text Detection. As mentioned above, CTW-1500 and Total-Text focus on text with arbitrary shapes, in which horizontal, multi-oriented and curved text exist simultaneously in most of images. Therefore, we use these two datasets to evaluate the effectiveness of TextFuseNet in detecting texts with arbitrary shapes. The last two columns of Table 2 lists the results of TextFuseNet compared to some previous methods on CTW-1500 and Total-Text, respectively. Note that FPS is only for reference, since different GPUs are adopted in different methods. As shown in Table 2, our proposed TextFuseNet using single-scale inference achieves state-of-the-art performance on both CTW-1500 and TotalText. Specifically, in CTW-1500, TextFuseNet with the backbone of ResNet-50 achieves an F-measure of 85.4%, higher than the current best one by 1.7%. When the backbone is ResNet-101, a more compelling result can be achieved (Fmeasure: 86.6%), outperforming all the other competitors by at least 2.9%. Similarly, for Total-Text, our TextFuseNet with ResNet-50 already achieves state-of-the-art result and its ResNet-101 version outperforms other approaches by at least 2.1%. The above experimental results show that TextFuseNet can access the state-of-the-art performance on arbitrary shape text detection.

Multi-oriented Text Detection. We also evaluated the effectiveness of TextFuseNet in detecting multi-orient text on ICDAR 2015. Our results and comparison with previous works are shown in the third column of Table 2. As Table 2 shows, TextFuseNet with backbone of ResNet-50 and ResNet-101 achieve state-of-the-art performance, and their Fmeasure are 90.1% and 92.1% respectively. Compared with the current best one, our ResNet-50 and ResNet-101 version outperform it by 2.4% and 4.4% respectively. Moreover, to the best of our knowledge, our proposed framework is the first one on ICDAR 2015 with an F-measure over 90.0%.

Horizontal Text Detection. Finally, we evaluated the effectiveness of TextFuseNet in detecting horizontal text on ICDAR 2013. The results of TextFuseNet and comparison with previous works are presented in the second column of Table 2. TextFuseNet with the backbone of ResNet-50 and ResNet101 both achieve very outstanding results, and the F-measure are 92.2% and 94.3% respectively, which outperform all previous works.

Therefore, from these experimental results on ICDAR 2013, ICDAR 2015, Total-Text and CTW-1500, our proposed TextFuseNet achieves state-of-the-art performance. In addition, for the speed, TextFuseNet can also inference at an appropriate speed, which has a degree of ascendancy compared to some previous methods. Some examples using TextFuseNet are shown in Figure 4.

5 Conclusion

In this paper, we propose a novel framework TextFuseNet for arbitrary shape text detection by investigating on three levels of features, i.e., character-, word- and global-level features. Different level features are fully and finely explored to learn richer fused features, which are beneficial to text detection. Our experimental results show that TextFuseNet achieves state-of-the-art performance in detecting text with arbitrary shape.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants (No.61822113, No.62041105), Australian Research Council Project (No.FL-170100117), the Natural Science Foundation of Hubei Province under Grants (No.2018CFA050) and the Science and Technology Major Project of Hubei Province (Next-Generation AI Technologies) under Grant (No.2019AEA170). The numerical calculations in this paper have been done on the supercomputing system in the Supercomputing Center of Wuhan University.