One-Shot Texture Retrieval with Global Context Metric

1 Introduction

As texture refers to the fundamental microstructures of natural image, humans have a strong visual perception of texture, which can not only obtains descriptions of new texture from a small number of training samples (few-shot learning) [Sung et al., 2018], but also marks such texture regions in the other cluttered scenes (texture segmentation) [Cimpoi et al., 2015]. This suggests that texture features provide a powerful visual prior for comprehensive scene understanding [Krishna et al., 2017].

To learn such texture prior, we present the problem of oneshot texture retrieval: given an example of a new reference texture, detect and segment all the pixels of the same texture category within an arbitrary image (see Figure 1). This task is different from one-shot segmentation of general objects [Shaban et al., 2017], as the learned texture representation should be invariant to spatial layout but preserve the rough semantic concepts. Therefore, an adaptable and robust texture encoding model should be presented to finely discriminate the Query Result orderless texture details. In addition, for texture segmentation, global context is also an important cue since the scene surfaces are usually not completely orderless. A similarity metric should be introduced to balance local spatial details and global scene context for pixel-wise segmentation.

∗Corresponding author

Figure 1: Examples of our task. Given a reference texture of new category, we can segment all pixels of the same texture category within a query image.

In this paper, we present an One-Shot Texture Retrieval (OS-TR) network to learn the texture representation, and model the similarities between reference and query image, leading to achieve one-shot texture segmentation towards the reference category. Specifically, a Siamese network [Koch et al., 2015] is first presented to embed the reference and query image into an encoded representation space by feature learning and parametric prior encoding. Unlike the existing texture encoding methods [Xue et al., 2018] that integrate CNN with orderless pooling, a directionality-aware module is proposed to perceive the local texture variations at each direction, resulting in spatially invariant representation. Then, we propose to incorporate global context into the relation network [Sung et al., 2018] by aggregating feature maps across their spatial dimensions. Different from previous approaches that only consider the similarity of local features or semantic concepts, our method exploits global context information to adjust local relation features by per-channel modulation weights. The key idea is to use a self-gating mechanism for generating global distribution of local relation features with an aggregation unit. The evaluation on synthetic dataset demonstrates the superiority of our proposed model against the state-of-the-art methods [Shaban et al., 2017; Ustyuzhaninov et al., 2018]. Furthermore, the result on nature scenes is promising.

Our main contribution are as follows:

1. We introduce a novel one-shot texture segmentation network with global context metric, achieving texture detection and segmentation with a single example of reference texture.
2. We propose a directionality-aware module to perceive the local texture variations at each direction, resulting in spatially invariant representation.
3. We present a global context metric for performing oneshot texture segmentation, which extends relation network with a self-gating mechanism to adjust local relation features.

2 RelatedWork

Our work focuses on one-shot learning, texture modeling and one-shot segmentation task, so in this section we mainly review the research status of these areas.

One-shot learning. In the computer vision community, one-shot learning has recently received a lot of attention and substantial progress has been made based on metric learning using Siamese neural networks [Koch et al., 2015; Snell et al., 2017]. In addition, there are some work that build upon meta learning [Finn et al., 2017; Ren et al., 2018], information retrieval techniques [Triantafillou et al., 2017] and generation models [Lake et al., 2015] to achieve one-shot learning.

Texture representation. Texture representation is an important research area in computer vision for potential applications in classification, segmentation and synthesis. The research of texture representation are mainly divided two classes: traditional method [Kumar et al., 2011] and CNNbased method [Cimpoi et al., 2015; Zhang et al., 2017]. Different from object recognition where spatial order is critical for feature representation, texture recognition usually uses an orderless component to provide invariance to spatial layout.

One-shot segmentation. While the work on one-shot learning is quite extensive, the research on one-shot segmentation [Dong and Xing, 2018; Wu et al., 2018] have been established only recently, including one-shot image segmentation [Shaban et al., 2017] and one-shot video segmentation [Caelles et al., 2018]. The most closely related to our work is [Ustyuzhaninov et al., 2018], whose task is to segment an input image containing multiple textures by given a patch of a reference texture. Different from their setup in that the reference patch is interactively selected from the input image, our work targets on a more complex problem of one-shot texture retrieval: given an example of a new reference texture, detect and segment all the pixels of the same texture category within an arbitrary image.

3 One-shot Texture Segmentation

3.1 Probelm Setup

We define a triple Trj i = (Qi,Rj i , T j i ) and a relation function F : Aji = F(Qi,Rj i ; θ), where Qi is the ith query image with multiple classes of textures, T j i is the pixel-wise labels corresponding to the j th class texture in Qi, Rj i is a reference texture image of the same class j, Aji is the actual segmentation result, and θ is all parameters to be optimized in function F. Our task is to randomly sample triples from the dataset, train and optimize θ, thus minimizing the loss function L:

We expect that the relationship function F can segment the same class texture region in another target image each time it sees a reference texture picture of new class. This is the embodiment of the meaning of one-shot segmentation. It should be mentioned that the texture classes sampled by the test set are not present in the training set, that is, Utrain ⋂ Utest = Ø. The relation function F in this problem is implemented by the model detailed in Section 3.2.

3.2 Model Architecture

In this section we will detail the overall framework of the proposed OS-TR network. As shown in Figure 2, the network is based on a classic encoder-decoder network [Ronneberger et al., 2015] to complete the segmentation task. It consists of the texture encoder for captioning the characteristics of the texture and global context metric that better determines the matching pixels. These two modules will be explained further in Section 3.3 and 3.4.

The network uses the texture encoders ft to transform the input of two branches into the variable embeddings respectively. The first branch takes the reference texture Rj i from the triple as input. And the second branch takes the synthetic texture image as input during training, which may be from a wide range of sources in the real scenarios. We use the texture encoder to perceive the local texture features at each dirction, and the corresponding feature map pairs M1, M2 obtained can be expressed as follows:

Here θt is the learnable parameter of our texture encoder.

Different from the existing work, one-shot relation network [Sung et al., 2018] proposes a deep nonlinear metric. However, this nonlinaer comparator is limited to local ralation features. Instead we learn a global context metric gc to compare the pixel information of query images with the reference texture by taking global context in consideration. It provides important features for subsequent pixel-level segmentation. In our network,

where S refers to the element-wise relation score, and θc is the parameter to be optimized in gc.

To generate the same size as the original image, we combine the score S with the information of encoder ft in multiple stages. We make full use of extracted features at different scales in the encoding process to form a refined decoding layer gd. The final actual segmentation result A of the original image is obtained through a sigmoid function:

Skip Connection

Similarly, θd stands for the parameters of decoding part.

The number of positive and negative samples in the training set is unbalanced (i.e., the foreground and background of query images). To ensure their equal contribution to optimization function, we use weighted Binary Cross Entory Loss function, that is

where yp stands for the ground truth of corresponding pixel p, α = |T-| /(|T-| + |T+|). T+ and T- denotes positive and negtive sample sets in training images, respectively. We can see θ in the loss function is the collection of θt, θc and θd described earlier. The purpose of reducing loss function is to optimize the parameters of corresponding modules.

3.3 Texture Encoder

To perceive the local texture variations at each direction, we propose the texture encoder consisting of feature extraction and directionality-aware module. As shown in Figure 2, ResNet (ResNet in this paper represents ResNet-50 from [He et al., 2016] that removes the last fully connected layer) is used to extract preliminary features P1, P2 from the query image and reference texture. Then the DirConv unit gd is presented to capture eight directional texture features Gi under the guidance of corresponding directional feature map Di. We only detail the first branch as the parameters and structures of two branches are identical. Specifically,

where gd denotes a convolution block which contains a 3 × 3 convolution, a batch normalization and a ReLU activation layer, cat refers to the concatenation function in the channel dimension. In this paper, eight directions refer to top, bottom, left, right, top left, bottom left, top right, and bottom right. Each directional map Dm is a generated trend graph that decreases from 1 to 0 along a certain direction. Finally, the output features of different branches are concatenated to form the whole spatial invariant feature M1, that is,

where gh represents a set of standard convolution block. Since the proposed DirConv unit is sensitive to the local variation of image along each direction, it can provide the network with good adaptability to spatial distortion and scale changes.

3.4 Global Context Metric

To achieve pixel-wise segmentation, we incorporate global context into local relation feature to adjust per-channel modulation weights. Firstly, we use a non-linear function gm to capture local relation features L:

where gm represents two sets of standard convolution blocks, H × W and C refers to spatial and channel dimensions of the feature map, respectively. It can be seen that the two-layer convolution module represents more metric possibilities, which is not limited by cosine, Euclidean distance [Vinyals et al., 2016; Snell et al., 2017], etc. However, it only considers local feature similarity which has its limitation. Instead we take global context [Qiao et al., 2019] into consideration by aggregating feature maps across their spatial dimensions, which is similar to SENet [Hu et al., 2018].

Let L = [l1, l2 · · · lC] (li ∈ RH×W ) denote the local relation features of different channels. In this paper, we simply obtain the global information of each channel β ∈ RC through maximum pooling gmax. The aggregation unit can be represented as follows:

Next we use the collected global context to balance the relation features. Here we use two simple 1 × 1 convolutional blocks gs to achieve per-channel modulation weights. Finally, the obtained weights are normalized to 0-1 and used as multiplication coefficient of corresponding channel of original feature L. The re-layout of local relation features can be formulated as follows:

where S is the same as the one in Equation 4 and · represents the channel-wise multiplication. It can be seen from the visualization of experimental part that global context metric realizes the fine adjustment of different texture matching pairs. It is beneficial for the optimization of the final segmentation result.

4 Experiments

To validate the superiority of our model in one-shot texture segmentation task, we designed a series of experiments based on Describable Textures Dataset (DTD) [Cimpoi et al., 2014] dataset. In this section, we first introduce the preprocessing process of the dataset, and then perform ablation experiments on main modules in the model. While demonstrating its superiority from objective indicators, we also give some good visualization results. Finally, we compare our model to the most advanced model One-Shot Learning for Semantic Segmentation (OSLSM) [Shaban et al., 2017] and One-shot Texture Segmentation (OSTC) [Ustyuzhaninov et al., 2018] in the current one-shot segmentation field. We also introduce expanded experimental contents in supplementary materials 1 .

4.1 Dataset and Setting

To solve the proposed one-shot texture segmentation task, we redivide DTD dataset into training and test sets. We divide the last 15 classes of DTD dataset into three test subsets on average, and the remaining classes are used as the training set. The specific class name is shown in Table 1. During the training, we randomly sample 2-5 texture images from the training set to synthesize query images (these textures may come from the same class), and generate labels for one of the texture regions. The technique of texture synthesis comes from [Ustyuzhaninov et al., 2018]. Finally, we randomly sample the reference texture image from the training set with the same kind of labels to form the triple defined in Section 3.1. In the test phase, we synthesize 240 query images (a total of 720 images) from the three subtest sets in the same way. We send them to the trained model to calculate evaluation indicators and then average them to ensure that different texture classes in the test set are relatively fair. The regular evaluation indicator IoU is used in our task.

1 https://github.com/zhukaii/OS-TR

Our model uses the SGD optimizer during the training process. The initial learning rate is set to 0.001 and the attenuation rate is set to 0.0005. The model stops training after 1000 epochs, where each epoch synthesizes 240 query images. All images are resized to 256 × 256 size and the batch size is set to 16.

4.2 Ablation Study

We conduct several ablation experiments to prove the effectiveness of directionality-aware module and global context metric of our model in Table 2. We first train a baseline without the two modules, and then add them respectively to compare the results. First of all, we can see that the global context metric improves the model by 2.6% mean IoU. This is due to its use of global information to help the model adjust the overall feature distribution. In order to explain this more vividly, we make a visualization of per-channel modulation weight distribution learned by global context metric in Section 4.3. Then, the result has a 2.5% mean IoU boost with the dierctionality-aware module. This module takes into account the characteristics of texture spatial distribution, which is very helpful for texture encoding. The specific analytical experiment is also shown in Section 4.3. Finally, we add both the two modules to form our OS-TR network, which has a 3.3% improvement over the baseline.

4.3 Evaluation

Spatial Invariance. To evaluate the performance of our model on the spatial invariant texture representation, we conduct the following experiments. First, in order to demonstrate the adaptability of our model to spatial distortion, we show its segmentation effect by affining the reference texture image. As shown in Figure 3 (A), we take several query images as examples and randomly determine the parameters of affine transformation to compare the segmentation results. It can be seen the results achieve the segmentation effect similar to the original reference texture. As shown in Figure 3 (B), we select three scale reference textures 256 × 256, 128 × 128 and 64 × 64 to evaluate the sensitivity of the model to scale change. The segmentation effect is still excellent. It is the proposed DirConv unit that is sensitive to the local variation of image along each direction, so that the spatial arrangement of texture can be accurately extracted.

Effectiveness of Global Context Metric. To demonstrate the function of the module, we visualize per-channel modulation weights for different reference images. We cite five categories in our test set as examples. As shown in two figures on the left in Figure 4, all the polylines of the same class OSTS

Our

Table 3: Mean IoU(%) of our model and other state-of-the-art methods.

in the left picture are almost identical and the right of five classes is different. To further explain this, we reduce the weights of these five classes and visualize them through tSNE [Maaten and Hinton, 2008]. It can be seen these five classes are clearly distinguished in the relation feature space. We think our module has learned an adaptive weight adjustment method among channels, where the intra-class is similar and inter-class appears inconsistent. As analyzed in Section

3.4, this module combines global information to further separate different matching pairs in feature space.

Comparision with state-of-the-art. To better assess the overall performance of our reference network, we compare it to OSLSM and OSTC models. Because the tasks and model backbones are different, we set all the backbones to ResNet for fair comparision, and reproduce them with pytorch according to the two articles. All models are trained and tested in the same steps to achieve the purpose of adapting to our task. OSLSM is the first solution to one-shot semantic segmentation task, which is similar to our task. So after modifying backbone, we train their models directly in our tasks and don't need to modify any dataset settings. To compare with OSTC, we change the reference image to a 64 × 64 patch for training in our task. As can be seen from Table 3, our model has more than three points mean IoU boost over the best performing OSLSM. Since OSTS is essentially designed for the interactive texture segmentation task, it does not work well for one-shot segmentation task. So we reproduce the model they use to solve their interactive texture segmentation task, which follows the settings of their papers. The reference texture of our model is the same class as a texture area in query image, and of course it may be the same texture image. Conversely, OSTS is not necessarily able to segment textures of the same class. As shown in Figure 5, the reference texture image in our model can achieve better segmentation effect whether it is the original image of a texture region of the query image or a different image of the same class. It benefits from the texture characteristics and the distinction of different texture features acquired by our texture module and global context metric respectively.

Evaluation on real scenarios. Our model not only performs well in synthetic texture images, but also has scalability in practical applications. To prove this, we replace the query iamges with ones taken from real scenarios. We download high-quality indoor scene pictures with rich texture information from OpenSurfaces dataset [Bell et al., 2013], and select animal and plant pictures with describable texture features from the Internet. For reference images, we randomly select images with similar texture information from DTD dataset and Internet according to query image content. For example, a striped image from DTD is randomly selected as a reference for zebras. In testing the above natural images, we did not pre-train or adjust the model. The parameters are still fixed in the state trained in synthetic texture images. As shown in Figure 6, We still get good segmentation results. It can be seen that our model can really learn new texture properties quickly, which is helpful for comprehensive scene understanding.

5 Conclusion

In this paper, a novel one-shot texture segmentation network OS-TR is proposed. By using a directionality-aware module to perceive texture variations at each direction and adjusting local features with global context information, OSTR extracts the pixel information related to given texture in query images. In addition, our model can be well generalized from synthetic images to real scenarios without any adjustment. Experimental results show that our model is superior in both performance and adaptability with respect to existing methods.

Acknowledgements

This work was supported by the National Key R&D Program of China under Grant 2017YFB130092, the National Natural Science Foundation of China (NSFC) under Grants 61872327, 61472380, 61622211 and 61620106009 as well as the Fundamental Research Funds for the Central Universities under Grant WK2380000001 and WK2100100030.