Consistent Domain Structure Learning and Domain Alignment for 2D Image-Based 3D Objects Retrieval

1 Introduction

The 3D objects retrieval, especially image-based 3D objects retrieval, is a very important task for data management. Most previous works aim to classify 3D objects or retrieve them within the same modality [Li et al., 2019b; He et al., 2018; Liu et al., 2018a]. For example, Li et al. [Li et al., 2019b] proposed an angular triplet-center loss for 3D retrieval and conducted retrieval experiments on the ModelNet dataset [Wu and et al., 2015] and ShapeNet dataset [Chang and et al., 2015], respectively.

However, there are less works about the 2D image-based 3D object retrieval with two-fold reasons: 1) the query and gallery are from different modalities and domains, and their features have different distributions, and 2) lack of related annotated datasets. Since it is of great value in application (e.g., 3D printing, entertainment) and research, some researchers and academic organizations pay attention to image-based 3D objects retrieval (e.g., [Lee et al., 2018; Li et al., 2019a]) and use domain adaptation to reduce domain gap [Su et al., 2019]. They often assume one domain as source domain with labels, and another one is target domain without labels. In this paper, we also adopt domain adaptation idea to reduce domain gap and learn a latent space for 2D images and 3D objects. We set 2D images as the source domain and 3D objects as the target domain.

∗Corresponding authors: Dan Song & An-An Liu

1.1 Motivations

2D image-based 3D objects retrieval aims to find related 3D objects in the gallery when given 2D images. There are two critical problems to solve for this promising but challenging task:

How to build a unified feature space for 2D images and 3D objects. 2D images and 3D objects are from different modalities. To reduce the gap between 3D objects and 2D images in visual level, we can capture a set of rendered images for each 3D object, e.g. MVCNN [Su et al., 2015]. In this case, we can extract the visual representations of 3D objects and 2D images by MVCNN and CNN (e.g., AlexNet [Krizhevsky et al., 2012]), respectively, and compute their similarity. However, the retrieval performances in such a naive way are poor with two-fold reasons: 1) their features have different distributions and 2) the high-dimensional features contain noisy information. Therefore, it is mandatory to explore a unified feature space with low-dimensional structure for 2D images and 3D objects.

How to fully exploit intrinsic structure of target domain for domain alignment. Domain alignment aims to reduce the discrepancy in marginal distributions (global level) and conditional distributions (local level). In unsupervised domain adaptation, the conditional distributions alignment at the local level plays a more important role than the marginal distributions alignment at the global level [Zhang et al., 2019]. To achieve the conditional distributions alignment, the most of existing methods just rely on the labeled samples in the source domain to train a classifier and predict the labels of samples from the target domain (e.g., [Su et al., 2019]). However, these methods neglect the intrinsic structure in the target domain and consequently result in poor quality of pseudolabels. Therefore, it is necessary to fully exploit semantic information from the source domain and the intrinsic structure in the target domain to better align the conditional distributions.

To handle the above issues, we propose a consistent domain structure learning and domain alignment framework to reduce the large domain gap for 2D images and 3D objects. As shown in Fig. 1, the framework contains five modules: visual feature extraction, subspace generation, consistent domain structure learning, domain alignment and retrieval. The visual feature extraction module aims to extract visual representation for 2D images and 3D objects. After this step, we use subspace generation module to remove noisy information in the original visual features and obtain more robust features. To learn a latent space for 2D images and 3D objects, we introduce a consistent domain structure learning (CDSL) module that can predict the class probability of the target domain, which further provides more reliable labels to the domain alignment module. At last, the output of the CDSL module, i.e. the class probability, is treated as the invariant feature for 2D images and 3D objects and similarity is measured by L2-distance.

1.2 Contributions

In this paper, the main contributions are as follows:

• We propose a novel visual domain adaptation method for 2D image-based 3D objects retrieval, which can learn a common latent subspace with domain alignment.
• We explore the domain structure consistency between 2D images and 3D objects to promote the prediction accuracy for the labels of the 3D objects domain, which can better guide the conditional distribution alignment.
• We conduct extensive experiments on two popular public datasets, MI3DOR and MI3DOR-2. The experimental results show the superiority of the proposed method.

2 RelatedWork

2.1 3DObjects Retrieval

3D objects retrieval has attracted more attentions than ever [Liu et al., 2016; 2018a; 2018b; Lu et al., 2019a; 2019b; Cheng et al., 2018; Hong et al., 2016]. Generally, we can divide existing 3D objects retrieval method into two categories: model-based methods and view-based methods.

Model-based methods [Xie et al., 2017; Shen et al., 2018] deal with the 3D structure of original objects, such as voxel grid [Ghadai et al., 2019] and point cloud [Qi et al., 2017]. For example, Ghadai et al. [Ghadai et al., 2019] learned features by a multi-level voxel representation of spatial data.

View-based methods represent each 3D object by a set of rendered images. For example, Su et al. [Su et al., 2015] proposed Multi-view Convolutional Neural Network (MVCNN) to represent 3D objects by aggregating the features of a set of rendered images in an end-to-end way. Feng et al. [Feng et al., 2018] modeled the hierarchical correlation to learn 3D shape description.

Currently, the image-based 3D objects retrieval framework is a new promising branch for 3D objects retrieval [Wang et al., 2015; Dai et al., 2017]. Dai et al. [Dai et al., 2017] reduced the discrepancy between sketch and 3D shape domain by a deep correlated metric learning method. However, the method focuses on sketch-based 3D objects retrieval. It is different from our task since natural images contain more complex backgrounds and more noisy objects when compared sketch. Recently, Lee et al. [Lee et al., 2018] learned a embedding space for images and 3D shapes in an end-to-end method. However, we can not annotate all the images and 3D

objects in practice. Therefore, it is necessary to reduce the cross domain gap by an unsupervised method.

2.2 Domain Adaptation

Generally, most of domain adaptation methods reduce the domain discrepancy in an unsupervised way, where given two related domains (sharing common classes), one labeled (called source domain) and another one unlabeled (called target domain).

At early time, unsupervised domain adaption methods focus on the marginal distribution alignment, e.g., Gopalan et al. [Gopalan et al., 2011] generated an intermediate representation between two domains on the Grassmann manifold. Later, Gong et al. [Gong et al., 2012] exploited lowdimensional structure with the GFK [Gopalan et al., 2011].

Although two domains are aligned in the marginal distributions, it can not guarantee to produce better classification performance. The main reason for this problem is the lack of labels of samples in the target domain and can not provide labels to align the conditional distributions. For example, Wang et al. [Wang et al., 2018] proposed an adaptive method to align the marginal and conditional distribution with pseudo-label learning. However, this method neglects the intrinsic structure of the target domain. Facing the large gap for our task, we adopt domain adaptation ideas to reduce domain discrepancy and learn invariant features for the query (2D images) and gallery (3D objects).

3 ProposedMethod

3.1 Overview

In this work, we propose a consistent domain structure learning and alignment framework. It can learn a projection matrix P that maps the visual representation of labeled source domain (2D images) Ds and unlabeled target domain (3D objects) Dt to a latent space of a lower dimensional Z. In the new learned common latent space Z, the domain structure consistency means: 1) nearby samples are likely from the same class; and 2) samples on the same structure (e.g., a cluster) are likely from the same class. Based on this assumption, we explore domain structure consistency to obtain high-quality pseudo labels for the target domain which benefits the conditional distribution alignment.

The proposed framework mainly contains four key modules (as shown in Fig. 1):

Visual representation. Visual representation module aims to obtain the representation of 3D objects or 2D images. For 3D objects, we can use a set of rendered images captured by cameras around the objects under different viewpoints. In this paper, we used Phong model [Phong, 1975] to capture the multi-view images for 3D objects and used MVCNN [Su et al., 2015] to obtain the visual representation of 3D objects. For 2D images, since 2D image can be as only one single view, we also used the same network to obtain the visual representation of 2D images.

Subspace generation. As we know, the high dimensional features always contain noisy information or redundant information. Therefore, we reduce the dimension of the feature first by Principle Component Analysis (PCA) and obtain more robust feature compared with the original feature data.

Domain alignment. Domain alignment is a key step for the domain adaptation. Previous works always rely on the distance between the feature points, while we suggest to use the label to constrain the distance based on whether the labels are the same (as shown in Eq.(3)).

Consistent domain structure learning. The consistent domain structure learning module aims to predict the label of the samples from target domain. It also provides more reliable likelihood information for the domain alignment. It is different from traditional pseudo-label learning methods since it joints the intrinsic structure of the target and source domains to promote the pseudo-labels prediction accuracy on the target domain.

Before introducing the details, we first define the notations. The 2D images belong to the source domain Ds = {(xsi , ys i )}( i = 1, 2, 3, ..., ns) and 3D objects belong to target domain Dt = {(xti )}( i = 1, 2, 3, ..., nt). xi ∈ Rd represents the visual representation of the ith sample from 2D images or 3D objects, d means the dimension of the original feature, and ys i ∈ Y s denotes the label of the ith sample in the source domain. Y means the label space. In this paper, we assume that Y t is equal to Y s .

3.2 Subspace Generation

The high dimensional feature space always contain noise information or redundant information. To address this, we adopt PCA to reduce the dimension of the feature first and obtain more robust feature compared with original feature data as in [Wang and Breckon, 2019]. We concatenated the features from the visual representation of 2D images and 3D objects as a matrix X = [xs1 , xs2 , · · · , xsns , xt1 , xt2 , · · · , xtnt ] ∈ Rd×n , where n = ns + nt. The objective of PCA is:

where H = I - 1n 1 is the center matrix, 1 denotes an all one n×n matrix, tr(·) means the trace of matrix. To solve Eq.(1), we transform Eq.(1) to the following eigenvalue problem:

Then by solving the Eq.(2), we can obtain the projection matrix V = [v1, · · · , vd1 ] ∈ Rd×d1 and the lower robust feature can be as ¯X = VT X, where ¯X ∈ Rd1×n . After applying PCA, the feature dimensionality is d1 and d1 ≤ d.

3.3 Domain Alignment

Subspace generation module aims to reduce noisy information in the original feature space and generate a new feature space ¯X in an unsupervised manner. However, it does not consider the domain discrepancy. Therefore, we hope we can find a projection matrix P , which can map ¯X to a more lower dimensional and discriminative space Z. In spired by MMD [Gretton et al., 2012] and locality preserving projection [He and Niyogi, 2003], we define the objection function as follows:

where P ∈ Rd1×d2 and d2 ≤ d1 is the learned feature space dimensionality. The term B ∈ R(ns+nt)×(ns+nt) , where Bij = 1 if yi = yj, otherwise Bij = 0, aims to choose good mapping for each class. The matrix B is a simple version of MMD. To align two domains and obtain the optimal P, we need to know the pseudo labels for the target objects. The pseudo label learning will be described in the next subsection.

3.4 Consistent Domain Structure Learning

The consistent domain structures learning module aims to predict the label of the samples from target domain in the common latent space Z and provides reliable likelihood information for domain alignment. We first define a probability annotation matrix M. Formally, let c ∈ {1, · · · ,C} represents the class label. The probability annotation matrix M ∈ RC×nt , where its entry value Mcj denotes the probability of xtj belonging to class c. The source domain data ¯xs and the target domain data ¯xt can be reformulated zs = PT ¯xs and zt = PT ¯xt , respectively.

Intrinsic Structure of the Source Domain

We use the class prototype of source domain as the intrinsic structure of the source domain. Each class center hsc of the source domain can be defined by the mean feature vector of the source domain samples whose label is c, which can be written as:

where I(·) denotes an indicator function. Then we can the conditional probability of a given target sample xt belonging to class c through the following equation:

Intrinsic Structure of the Target Domain

After finishing the above step, we can obtain the labels of samples in the target domain. However, the above method does not consider the intrinsic structure of the target domain. To explore such structure information, we use K-means to generate |C| clusters in the projection feature space Z.

We first initialize the cluster center of the target domain by Eq.(4). Then, to make the cluster centroid of the target domain close to the class prototype of the source domain, we define the optimization problem as in [Wang and Breckon, 2019]:

where A ∈ {0, 1}|C|×|C| represents a matching matrix where Aij = 1 means the ith center of the target cluster matches with the j th the class center of source domain. hti denotes the ith cluster center in the target domain. The Eq.(6) can be solved according to [Wang and Chen, 2017].

Let htc denotes the cluster center corresponding to the class c, similar to Eq. (5), we can calculate the conditional probability of a given target sample xt belonging to class c:

Consistent Structure Learning

Although the above two methods can label the samples from the target domain by iterative learning strategy, they are intrinsically different. The high probability p1(c|xt ) means the corresponding samples are close to the class center of the source domain, whilst the high probability p2(c|xt ) means the corresponding samples are close to the cluster center in the target domain regardless of the distance to the source domain. To make use of the these complementary information, we combine Eq.(5) and Eq.(7) as follows:

where µ is a trade-off parameter. As a result, the pseudolabel of a given target sample xt can be predicted by: ¯ yt = argmaxc∈Y p(c|xt ). Now, we obtain all the pseudo-labels of samples in the target domain and compute the optimum P.

3.5 Optimization

The objective Eq.(3) can be rewritten as:

where L = D - B is the laplacian matrix, and D denotes a diagonal matrix whose element Dii = ∑j Bij. ̂X ∈ Rd1×(ns+nt) is the combination matrix for all data. For Eq.(9), we add a regularization term Tr(PT P) to obtain a better projection matrix P. Therefore, the Eq.(9) can be rewritten as:

Eq.(10) is equivalent to the following generalized eigenvalue problem:

To solve Eq.(11), we set P = [p1, p2, · · · , pd2 ], where p1, p2, · · · , pd2 are eigenvectors in terms of the largest d2 eigenvalues.

4 Experiment

4.1 Datasets

MI3DOR. The dataset [Li et al., 2019a] has 21 categories and contains 21,000 images and 7690 3D objects. The 2D training set is about 10500 images and the rest images are be used for testing. The 3D objects training set contains 3,842 objects and the rest 3,748 objects are used for testing.

MI3DOR-2. The dataset [Su et al., 2019] has 40 categories and also contains 19,694 images and 3,982 objects. The 2D training set is about 19,294 images and the rest images are used to test. For the 3D objects, 3,182 objects are used for training and the rest 800 objects are used for testing.

4.2 Evaluation Criteria & Implementation Details

We adopted NN, FT, ST, DCG, F-Measure and ANMRR as the evaluation criteria, as described in [Liu et al., 2018a]. These criteria range from 0 to 1, with the exception of ANMRR, the higher the better.

To obtain the rendered images for 3D objects, we set the cameras every 30 degrees around each 3D object and then obtain 12 view images for it. In our experiments, we used the AlexNet as CNN architecture in the MVCNN. The dimension of the original visual feature of 2D images and 3D objects is d = 4096. The dimension of subspace obtained by PCA is d1 = 512, and the dimension of the latent space obtained by P is d2 = 128. We set the number of iterations T = 6.

4.3 Comparison with State-of-the-Art Methods

We compared the proposed method with several representative methods: 1) 1NN means using the original visual representation of 2D images and 3D objects before the subspace generation to measure their similarity; 2) CORAL [Sun et al., 2016] aligns the second-order statistical characteristics of the source and target domains; 3) GFK [Gong et al., 2012] integrates unlimited number of subspaces to reduce domain shift; 4) JGSA [Zhang and et al., 2017] aligns distributions by the label propagation; 5) JAN [Long et al., 2017] aligns the distributions in a transfer network; 6) LSR [Xie et al., 2018] aligns the centroids of labeled source domain and the centroids of pseudo-labeled target domain to learn semantic representations of the samples from the target domain ; 7) JHFL [Su et al., 2019] can adaptively align the distributions for 2D images and 3D objects.

The comparison is shown in Table 1. From the results, we can obtain four key observations.

1) Our method is better than representative methods since it contains consistent domain structure learning, which benefits the domain alignment. On MI3DOR, the proposed method can achieve the gain of 14.0%-222.1% in NN, 1.1%-337.8% in FT, 20.0%-233.7% in ST, 10.2%-253.6% in F-measure and 3.2%-321.6% in DCG, and decline 4.3%-60.9% in ANMRR.

• On MI3DOR-2, the proposed method is better than the representative methods except JHFL. When compared with JHFL, the proposed method achieves better in NN and ST.
• JGSA and JHFL are significantly better than CORAL and GFK. This reveals that aligning both the marginal distributions and conditional distributions is more important than only aligning the marginal distributions.
• All methods except for 1NN are better than 1NN and it demonstrates there exists a large gap between 2D and 3D visual representations.
• The proposed method is better than JAN and LSR. JAN and LSR require large scale data and tuning a large number of parameters to obtain the best results. In contrast, the proposed method can set fewer parameters with only cross-validation and a small amount of data.

4.4 Ablation Study

For the proposed method, the consistent domain structure learning module is a very important module. To evaluate the importance of the class centers from the source domain (marked, CCS) or the intrinsic structure of the target domain (marked as IST) in the consistent domain structure learning module, we design and conduct three comparison experiments: CCS, IST, CCS with IST (marked as CCS+IST).

In Table 2, we obtain two key observations: 1) The semantic information from the source domain plays a key role to predict soft label. Especially, CCS can significantly improve the performances when compared with '1NN' in Table 1. 2) The IST can be as a complementary information for the CCS and benefits the retrieval performances.

4.5 Effects of Hyper-parameters

There are four hyper-parameters in the proposed method: the trade-off parameter µ for the intrinsic structure learning module, the dimensionality of subspace d1 after PCA, the latent space d2 obtained by the projection P and the number of iterations T . We evaluate them on MI3DOR and MI3DOR-2. The results are shown in Fig. 2.

µ is a very important parameter to the consistent domain structure learning module. As shown in Fig. 2 (a) and (e), the optimal µ is 0.01 for MI3DOR and 0.1 for MI3DOR-2. The d1 and d2 are also very important parameters for our task. On MI3DOR, as shown in Fig. 2 (b) and (c), the optimal d1 and d2 is 512 and 64, respectively. On MI3DOR-2, as shown in Fig. 2 (f) and (g), the optimal d1 and d2 is 512 and 128, respectively. In Fig. 2 (d) and (h), the performance gains slowly with iterations growing. The optimal T is 15 and 6 on MI3DOR and MI3DOR-2, respectively. To summarize, the proposed method is robust to these hyper-parameters except d2, which should be larger than the number of classes.

5 Conclusion

In this paper, we propose a consistent domain structure learning and domain alignment framework for 2D image-based 3D objects retrieval. It can reduce domain discrepancy and learn a latent feature space of a lower dimension. The consistent domain structure learning module explores both the semantic information of the source domain and the intrinsic structure of the target domain. It can provide reliable labels for the unlabeled target domain to domain alignment. The extensive experiments show that the proposed method achieve remarkable results on two public datasets.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (61772359, 61872267, 61902277, 61572356), the grant of Tianjin New Generation Artifificial Intelligence Major Program(19ZXZNGX00110, 18ZXZNGX00150, 17ZXRGGX00180), the Open Project Program of the State Key Lab of CAD & CG, Zhejiang University (Grant No. A2005, A2012).