Hi-Fi: Hierarchical Feature Integration for Skeleton Detection

1 Introduction

Object skeletons are defined as the medial axis of foreground objects surrounded by closed boundaries [Blum, 1967]. Complementary to object boundaries, skeletons are shape-based descriptors which provide a compact representation of both object geometry and topology. Due to its wide applications in other vision tasks such as shape-based image retrieval [Demirci et al., 2006; Sebastian et al., 2004], and human pose estimation [Girshick et al., 2011; Shotton et al., 2013; Sun et al., 2012]. Skeleton detection is extensively studied very recently [Shen et al., 2017; Ke et al., 2017; Tsogkas and Dickinson, 2017].

Because the skeleton scales (thickness) are unknown and may vary among objects and object parts, skeleton detection has to deal with a more challenging scale space problem [Shen et al., 2016b] compared with boundary detection, as shown in Fig. 1. Consequently, it requires the detector to capture broader context for detecting potential large-scale (thick) skeletons, and also possess the ability to focus on local details in case of small-scale (thin) skeletons.

∗M.M. Cheng is the corresponding author.

Performing multi-level feature fusion has been a primary trend in pixel-wise dense prediction such as skeleton detection [Ke et al., 2017] and saliency detection [Zhang et al., 2017; Hou et al., 2018]. These methods fuse CNN features of different levels in order to obtain more powerful representations. The disadvantage of existing feature fusion methods is that they perform only deep-to-shallow refinement, which provides shallow layers the ability of perceiving high-level concepts such as object and image background. Deeper CNN features in these methods still suffer from low-resolution, which is a bottleneck to the final detection results.

In this paper we introduce hierarchical feature integration (Hi-Fi) mechanism with bidirectional guidance. Different from existing feature fusing solutions, we explicitly enable both deep-to-shallow and shallow-to-deep refinement to enhance shallower features with richer semantics, and enrich deeper features with higher resolution information. Our architecture has two major advantages compared with existing alternatives:

BidirectionalMutual Refinement. Different fromexisting solutions illustrated in Fig. 2 (a) and Fig. 2 (b) where dif-

�D�

�E�

�F�

�G�

Figure 2: Different multi-scale CNN feature fusing methods: (a) side-outputs as independent detectors at different scales [Xie and Tu, 2015; Shen et al., 2016b]; (b) deep-to-shallow refinement [Ke et al., 2017] that brings high-level semantics to lower layers; (c) directly fuse all feature levels at once; (d) our hierarchical integration architecture, which enables bidirectional mutual refinement across low/high level features by recursive feature integration.

ferent feature levels work independently or allow only deepto-shallow guidance, our method Fig. 2 (d) enables not only deep-to-shallow but also shallow-to-deep refinement, which allows the high-level semantics and low-level details to mutually help each other in a bidirectional fashion.

Hierarchical Integration. There are two alternatives for mutual integration: directly fusing all levels of features as shown in Fig. 2 (c), or hierarchically integrating them as shown in Fig. 2 (d). Due to the significant difference between faraway feature levels, directly fusing all of them might be very difficult. We take the second approach as inspired by the philosophy of ResNet [He et al., 2016], which decomposes the difficult problem into much easier sub-problems: identical mapping and residual learning. We decompose the feature integration into easier sub-problems: gradually combining nearby feature levels. Because optimizing combinations of close features is more practical and easier to converge due to their high similarity. The advantage of hierarchical integration over directly fusing all feature levels is verified in the experiments (Fig. 9).

2 RelatedWork

SkeletonDetection. Numerousmodels have been proposed for skeleton detection in the past decades. In general, they can be divided into three categories: (a) early image processing based methods, these methods localize object skeleton based on the geometric relationship between object skeletons and boundaries; (b) learning based methods, by designing handcrafted image features, these methods train a machine learning model to distinguish skeleton pixels from non-skeleton pixels; (c) recent CNN-based methods which design CNN architectures for skeleton detection.

Most of the early image processing based methods [Morse et al., 1993; Jang and Hong, 2001] rely on the hypothesis that skeletons lie in the middle of two parallel boundaries. A boundary response map is first calculated (mostly based on image gradient), then skeleton pixels can be localized with the geometric relationship between skeletons and boundaries. Some researchers then investigate learning-based models for skeleton detection. They train a classifier [Tsogkas and Kokkinos, 2012] or regressor [Sironi et al., 2014] with hand-crafted features to determine whether a pixel touches the skeleton. Boundary response is very sensitive to texture and illumination changes, therefore image processing based methods can only deal with images with simple backgrounds. Limited by the ability of traditional learning models and representation capacity of hand-crafted features, they cannot handle objects with complex shapes and various skeleton scales.

More recently many researchers have been exploiting the powerful convolutional neural networks (CNNs) for skeleton detection and significant improvements have been achieved on several benchmarks. HED [Xie and Tu, 2015] introduces side-output that is branched from intermediate CNN layers for multi-scale edge detection. FSDS [Shen et al., 2016b] then extends side-output to be scale-associated side-output, in order to tackle the scale-unknown problem in skeleton detection. The side-output residual network (SRN) [Ke et al., 2017] exploits deep-to-shallow residual connections to bring high-level, rich semantic features to shallower side-outputs with the purpose of making the shallower side-outputs more powerful to distinguish real object skeletons from local reflective structures.

Multi-Scale Feature Fusing in CNNs. CNNs naturally learn low/mid/high level features in a shallow to deep layer fashion. Low-level features focus more on local detailed structures, while high-level features are rich in conceptual semantics [Zeiler and Fergus, 2014]. Pixel-wise dense prediction tasks such as skeleton detection, boundary detection and saliency detection require not only high-level semantics but also high-resolution predictions. As pooling layers with strides down-sample the feature maps, deeper CNN features with richer semantics are always with lower resolution.

Many researchers [Ke et al., 2017; Zhang et al., 2017; Hou et al., 2018] try to fuse deeper rich semantic CNN features with shallower high-resolution features to overcome this semantic vs resolution conflict. In SRN, Ke et. al. [2017] connected shallower side-outputs with deeper ones to refine the shallower side-outputs. As a result, the shallower sideoutputs become much cleaner because they are capable of suppressing non-object textures and disturbances. Shallower side-outputs of methods without deep-to-shallow refinement such as HED [Xie and Tu, 2015] and FSDS [Shen et al., 2016b] are filled with noises.

A similar strategy has been exploited in DSS [Hou et al., 2018] and Amulet [Zhang et al., 2017] for saliency detection, and a schema of these methods with deep-to-shallow refinement can be summarized as Fig. 2 (b). The problem of these feature fusion methods is that they lack the shallowto-deep refinement, the deeper side-outputs still suffer from low-resolution. For example, DSS has to empirically drop the last side-output for its low-resolution.

3 Hi-Fi: Hierarchical Feature Integration

3.1 Overall Architecture

We implement the proposed Hi-Fi architecture based on the VGG16 [Simonyan and Zisserman, 2015] network, which has 13 convolutional layers and 2 fully connected layers. The conv-layers in VGG network are divided into 5 groups: conv1-x, ..., conv5-x, and there are 2∼3 conv-layers in a group. There are pooling layers with stride = 2 between neighbouring convolution groups.

In HED, the side-outputs connect only with the last convlayer of each group. RCF (Richer Convolution Features) [Liu et al., 2017] connects a side-output to all layers of a convolutional group. We follow this idea to get more powerful convolutional features. The overall architecture of Hi-Fi is illustrated in Fig. 3, convolutional groups are distinguished by colors, and pooling layers are omitted.

3.2 Hierarchical Feature Integration

A detailed illustration of the proposed feature integration procedure is shown in Fig. 4. Feature maps to be integrated are branched from the primary network stream through a (1 × 1) convolutional layer (dotted boxes marked with (a)) on top of an interior convolutional layer. These feature maps are further integrated with element-wise sum (box marked with (c)). The final scale-associated side-output (box marked with (d)) is produced by a (1×1) convolution. Note that due to the existence of pooling layers, deeper convolutional feature maps are spatially smaller than shallower ones. Upsampling (box marked with (b)) is required to guarantee all feature maps to be integrated are of the same size.

Ideally, the feature integration can be recursively performed until the last integrated feature map contains information from all convolution layers (conv1-1 ∼ conv5-3). However, limited by the memory of our GPUs and the training time, we end up with two level integration (Fig. 3 'Hi-Fi-2').

Bidirectional Refinement. We explain the proposed bidirectional mutual refinement by comparing it with existing architectures: FSDS [Shen et al., 2016b] and SRN [Ke et al., 2017]. As shown in Fig. 5, side-outputs (SOs) of FSDS are working independently, there is no cross talk between features of different levels. As a result, FSDS has noisy shallow SOs and low-resolution deeper SOs. SRN then introduces deep-to-shallow refinement by bringing deep features to shallow SOs. As shown in Fig. 5, shallower SOs of SRN are much cleaner than that of FSDS. Despite the improvement, deeper SOs in SRN are still suffering from low-resolution, which limits the quality of the final fused result.

In our architecture SOs are built on top of an integration of nearby feature levels, and the "nearby feature integration" is recursively performed. In testing phase, SOs will receive information from both deeper and shallower sides; and in training phase, gradient from SOs will back-prop to both as well. In other words, our approach explicitly enables not only deep-to-shallow but also shallow-to-deep refinement. It is obviously shown in Fig. 5 that Hi-Fi obtains cleaner shallower SOs than FSDS, and at the same time has much more highresolution deeper SOs than SRN. Consequently, we gain a strong quality improvement in the final fused result.

3.3 Formulation

Here we formulate our approach for skeleton detection. Skeleton detection can be formulated as a pixel-wise binary classification problem. Given an input image X = {xj, j = 1, ..., |X|}, the goal of skeleton detection is to predict the corresponding skeleton map ˆ Y = { ˆ yj, j = 1, ..., |X|}, where

ˆ yj ∈ {0, 1} is the predicted label of pixel xj. ˆ yj = 1 means pixel xj is predicted as a skeleton point, otherwise, pixel xj is the background.

Ground-truth Quantization. Following FSDS [Shen et al., 2016b], we supervise the side-outputs with 'scaleassociated' skeleton ground-truths. To differentiate and supervise skeletons of different scales, skeletons are quantized into several classes according to their scales. The skeleton scale is defined as the distance between a skeleton point and its nearest boundary point. Assume S = {sj, j = 1, ..., |X|}(sj ∈ R) is the skeleton scale map, where sj represents the skeleton scale of pixel xj. When xj is the background, sj = 0. Let Q = {qj, j = 1, ..., |X|} be the quantized scale map, where qj is the quantized scale of pixel xj. The quantized scale qj can be obtained by:

where rm (m = 1, ...,M) is the receptive field of the m-th side-output (SO-m), with r0 = 0, and M is the number of side-outputs. For instance, pixel xj with scale sj = 39 is quantized as qj = 3, because 14 = r2 < sj ⩽ r3 = 40 (receptive fields of side-outputs are shown in Fig. 3 with numbers-in-boxes). All background pixels (sj = 0) and skeleton points out of scope of the network (sj > rM) are quantized as 0.

Supervise the Side-outputs. Scale-associated side-output is only supervised by the skeleton with scale smaller than its receptive field. We denote ground-truth as Gm = {gm j , j = 1, ..., |X|}, which is used to supervise SO-m. Gm is modified fromQwith all quantized values larger than m set to zero. gm j can be obtained from Q by:

Supervising side-outputs with the quantized skeleton map turns the original binary classification problem into a multiclass problem.

Fuse the Side-outputs. Suppose P m = {pmj,k , j = 1, ..., |X|, k = 0, ...,Km} is the predicted probability map of SO-m, in which Km is the number of quantized classes SO-m can recognise. pmj,k means the probability of pixel xj belonging to a quantized class #k, and index j is over the spatial dimensions of the input image X. Obviously we have ∑Km k=0 pmj,k = 1.

We fuse the probability maps from different side-outputs P m(m = 1, ...,M) with a weighted summation to obtain the final fused prediction P = {pj,k}:

where pj,k is the fused probability that pixel xj belongs to quantized class #k, wmk is the credibility of side-output m on quantized class #k. Eq. (3) can be implemented with a simple (1 × 1) convolution.

Loss Function and Optimization. We simultaneously optimize all the side-outputs P m(m = 1, ...,M) and the fused prediction P in an end-to-end way. In HED [2015], Xie and Tu introduce a class-balancing weight to address the problem of positive/negative unbalancing in boundary detection. This problem still exists in skeleton detection because most of the pixels are background. We use the class-balancing weight to offset the imbalance between skeletons and the background. Specifically, we define the balanced softmax loss l(P, G) as:

where P = h(X|Θ) is the prediction from CNN, G is the ground-truth, and m is the index of side-outputs. h(X|Θ) is the model hypothesis taking image X as input, parameterized by Θ. βm = [∑|X| j=1 1(gm j ̸= 0)] /|X| is a balancing factor, where 1(·) is an indicator. The overall loss function can be expressed as follow:

where P m and Gm are prediction/ground-truth of SO-m respectively, G is the ground-truth of final fused output P .

All the parameters including the fusing weight wmk in Eq. (3) are part of Θ. We can obtain the optimal parameters by a standard stochastic gradient descent (SGD):

Detect Skeleton with Pretrained Model. Given trained parameters Θ, the skeleton response map ˆ Y = {ˆ yj, j = 1, ..., |X|} is obtained via:

where ˆ yj ∈ [0, 1] indicates the probability that pixel xj belongs to the skeleton.

4 Experiments and Analysis

In this section, we discuss the implementation details and report the performance of the proposed method on several open benchmarks.

4.1 Datasets

The experiments are conducted on four popular skeleton datasets: WH-SYMMAX [Shen et al., 2016a], SKSMALL [Shen et al., 2016b], SK-LARGE 1 [Shen et al., 2017] and SYM-PASCAL [Ke et al., 2017]. Images of these datasets are selected from other semantic segmentation datasets with human annotated object segmentation masks, and the skeleton ground-truths are extracted from segmentations. Objects of SK-SMALL and SK-LARGE are cropped from MSCOCO dataset with 'well defined' skeletons , and there is only one object in each image. SYM-PSCAL selects images from the PASCAL-VOC2012 dataset without cropping and here may be multiple objects in an image.

Some representative example images and corresponding skeleton ground-truths of these datasets are shown in Fig. 6.

(a) SYM-PASCAL (b) SK-SMALL

(c) SK-LARGE (d) WH-SYMMAX

Figure 6: Example images and corresponding skeleton groundtruths (red curves) of several skeleton datesets.

4.2 Implementation Details

We implement the proposed architecture based on the openly available caffe [Jia et al., 2014] framework. The hyperparameters and corresponding values are: base learning rate

1 http://kaiz.xyz/sk-large

(10-6 ), mini-batch size (1), momentum (0.9) and maximal iteration (40000). We decrease the learning rate every 10,000 iterations with factor 0.1.

We perform the same data augmentation operations with FSDS for fair comparison. The augmentation operations are: (1) random resize images (and gt maps) to 3 scales (0.8, 1, 1.2), (2) random left-right flip images (and gt maps); and (3) random rotate images (and gt maps) to 4 angles (0, 90, 180, 270).

4.3 Evaluation Protocol

The skeleton response map ˆ Y is obtained through Eq. (7), to which a standard non-maximal suppression (NMS) is then applied to obtain the thinned skeleton map for evaluation. We evaluate the performance of the thinned skeleton map ˆ Y in terms of F-measure= 2 Precision·Recall Precision+Recall as well as the precision recall curve (PR-curve) w.r.t ground-truth G. By applying different thresholds to ˆ Y , a series of precision/recall pairs are obtained to draw the PR-curve. The F-measure is obtained under the optimal threshold over the whole dataset.

4.4 Skeleton Detection

We test our method on four aforementioned datasets. Example images and ground-truths of these datasets are shown in Fig. 6, some detection results by different approaches are shown in Fig. 8. Similar to RCF [Liu et al., 2017] we perform multi-scale detection by resizing input images to different scales (0.5, 1, 1.5) and average their results. We compare the proposed approach with other competitors including one learning based method MIL [Tsogkas and Kokkinos, 2012], and several recent CNN-based methods: FSDS [Shen et al., 2016b], SRN [Ke et al., 2017], HED [Xie and Tu, 2015], RCF [Liu et al., 2017]). FSDS and SRN are specialized skeleton detectors, HED and RCF are developed for edge detection. Quantitative results are shown in Fig. 7 and Tab.1, our proposed method outperforms the competitors in both terms of F-measure and PR-curve. Some representative detection results are shown in Fig. 8.

Comparison results in Tab. 1 reveal that with 1st-level hierarchical integration (Hi-Fi-1), our method (Hi-Fi-1) already outperforms others with a significant margin. Moreover, by integrating 1st-level integrated features we obtain the 2ndlevel integration (Hi-Fi-2) and the performance witnesses a further improvement (Limited by the GPU memory, we didn't implement Hi-Fi-3). Architecture details of Hi-Fi-1 and HiFi-2 are illustrated in Fig. 3.

'square-shaped' objects like monitors and doors whose skeletons are hard to define and recognise.

5 Conclusion

We propose a new CNN architecture named Hi-Fi for skeleton detection. Our proposed method has two main advantages over existing systems: (a) it enables mutual refinement with both deep-to-shallow and shallow-to-deep guidance; (b) it recursively integrates nearby feature levels and supervises all intermediate integrations, which leads to a faster convergence and better performance. Experimental results on several benchmarks demonstrate that our method significantly outperforms the state-of-the-arts with a clear margin.

Acknowledgments

This research was supported by NSFC (NO. 61620106008, 61572264, 61672336), Huawei Innovation Research Program, and Fundamental Research Funds for the Central Universities.