Hierarchical Instance Feature Alignment for 2D Image-Based 3D Shape Retrieval

1 Introduction

1.1 Motivation

3D shape retrieval aims to match the similar shapes in the gallery given a query object, which can be the sketch [Li and et al., 2012], 2D image [Zhou et al., 2019a] or 3D shape [Zhou et al., 2019b]. Because of its significant applications in 3D printing, digital entertainment, medical engineering and computer aided design [Lu et al., 2019a; 2019b;

∗Corresponding authors: Weizhi Nie & An-An Liu

Cheng et al., 2018; Hong et al., 2016], 3D shape retrieval has attracted much attention from both industry and academic fields. Although deep learning has been adopted for 3D shape retrieval community and achieved significant performances, few literature focuses on 2D image-based 3D shape retrieval, which has not been well studied to date. They still yield weak performances caused by the following critical problems:

Difficulty in narrowing the gap between the 2D and 3D domains without strong dependence on annotated 3D shapes. Most existing 3D shape retrieval methods employ supervised learning and are usually based on learning multiview context [Zhou et al., 2019b], learning discriminative features from point cloud [Qi et al., 2017a] or discovering the structure information from voxel [Maturana and Scherer, 2015]. These methods rely on substantial annotated 3D shapes while manually labeling may be unreliable and is time-consuming, which limits the practicability and usability of the supervised learning methods for the real applications. [Liu et al., 2018] focuses on the unsupervised feature learning to solve this problem by transferring the knowledge from the label-rich 3D domain to the unlabled 3D domain. However, this method is based on the assumption that there must exist a common subspace between the two domains. This assumption can not hold for 2D image-based 3D shape retrieval task, because the gap between 2D and 3D is significant due to the background complexity, light intensity, viewpoint variance and modality discrepancy. Therefore, it's mandatory to develop sophisticated algorithms to narrow the gap between the 2D and 3D domains without annotated 3D data for retrieval.

Difficulty in enforcing the alignment of global domain statistics and local semantic information for cross-modal data. 2D image-based 3D shape retrieval is challenging mainly due to the great global domain and local semantic variations. By global domain variation we mean that 3D shape and 2D image are essentially heterogeneous since the data formats are completely different as the basic unit of the 2D image and 3D shape are pixel and voxel, respectively. By local semantic variation we mean the class-level difference across two domains including background, resolution, size, lightness, occlusion, viewpoint, etc. can be significant even when we describe the shape by multi-view images. However, the current domain adaptation methods only focus on Stairs the global domain statistics alignment by confusing the domain discriminator or minimizing the domain-level distances, such as CORAL [Long et al., 2013], MMD [Borgwardt et al., 2006], RevGard [Ganin and Lempitsky, 2015] and so on. As shown in Figure. 1, the gap between two domains may be difficult to align in the local semantic level even with the well domain confusion. Thus, it is urgently desired to design a 2D image-based 3D shape retrieval method with both global domain statistics and local semantic information alignment.

Figure 1: Some 2D image and 3D shape samples on MI3DOR.

Difficulty in learning the discriminative instance feature for both 2D image and 3D shape. Although deep learning has achieved great progress on feature learning for both 2D image and 3D shape, it only focuses on the correlations between the encoder output and the label while neglecting the connections between the output feature and the input. Specifically, for different instances of the same class, the existing methods tend to extract the common-class features while ignoring the unique visual characteristics of individual instances, which may have negative influence on the performance. For examples, in Figure.1, there are two classes, bed and stairs, which have five different instances in each domain, respectively. The common features of both classes are not discriminative enough to keep the visual details clearly, which may weaken the ability of feature representation. Therefore, it's necessary to learn the discriminative instance feature for preserving as much visual characteristics of the cross-data as possible.

To overcome these problems, we propose a novel hierarchical instance feature alignment (HIFA) network for 2D imagebased 3D shape retrieval. As shown in Figure. 2, we first employ the identical CNN to extract the visual features for both 2D image and 3D shape (a set of multi-view images) with the max-pooling operation [Su et al., 2015]. To preserve the instance visual characteristics, we maximize the mutual information between the 2D image / 3D shape and the high-level features. To narrow the gap between the two domains, we propose the hierarchical instance feature alignment module to enforce both global and local alignment. Since domain shift is decided by misaligned feature-norm expectations [Xu et al., 2019], we impose the identical large norm restriction on both the 2D and 3D feature-norm expectations to facilitate more transferable possibility. However, well domain alignment is not enough for retrieval since the semantic information for each class may be neglected. We further propose to eliminate the class-level variations by minimizing the distance between the two centroids of the same class from different domains.

1.2 Contributions

In summary, the main contributions of this paper as follows:

• We propose an unsupervised 2D-image based 3D shape retrieval method, which can jointly eliminate the global domain and local semantic variations by the hierarchical instance feature alignment module.
• Different from the existing 3D shape retrieval methods concentrating on learning the common-class features, we first propose to learn the discriminative instance feature, which can preserve as much as visual characteristics for retrieval task.
• Experimental analysis on two widely used datasets, MI3DOR and MI3DOR-2, demonstrates that our method can outperform the state-of-the-art 2D imagebased 3D shape retrieval approaches.

2 RelatedWork

2.1 3D Shape Retrieval

The typical 3D shape retrieval methods mainly includes viewbased methods and model-based methods. View-based methods usually describe 3D shape by the multi-view image set and encode individual views by the 2D CNN. [Su et al., 2015] proposed Multi-view Convolutional Neural Networks (MVCNN), which fused the multi-view features by a max-pooling layer across different views. Unlike the viewto-shape setting in MVCNN, [Feng et al., 2018] proposed a view-group-shape architecture, group-view convolutional neural networks (GVCNN), which divided views into several clusters based on visual similarity and then fused their features in both group-level and shape-level. Model-based methods usually directly extract the features from origin 3D data, such as point cloud and voxel. [Qi et al., 2017a] proposed PointNet, which first applied CNN to deal with unordered point sets. However, PointNet can not capture the local structures of the point sets, which resulted in the less discriminative of the feature representation. To resolve this problem, [Qi et al., 2017b] further proposed PointNet++, which utilized the neighbor points at multi-scale to capture the local structure information. [Maturana and Scherer, 2015] proposed VoxNet. This method utilized supervised convolutional network to encode the binary voxel grids for 3D shape representation.

2.2 Domain Adaptation

Domain adaptation aims to narrow the gap between two data distributions, usually with unlabeled target data and labeled source data. A common strategy for domain adaptation is to learn domain-invariant features. For example, [Long et al., 2013] proposed joint distribution adaptation (JAN) network, which aimed to minimize the joint MMD on the specific domain adaptation layer across two domains to learn the domain-invariant features. [Ganin and Lempitsky, 2015] proposed domain adversarial neural network (DANN). This method imposed an additional domain discriminator to the traditional CNN network to enforce the domain alignment. Based on DANN, which only narrowed the domain shift in the global distribution statistics level, [Xie et al., 2018] proposed to align the feature centroids of the same class from different domains to preserve the semantic consistency in local class level. [Wang et al., 2018] focused on learning a domain-invariant classifier instead of feature in grassmann manifold with structural risk minimization to perform distribution alignment. [Xu et al., 2019] enabled the CNN network to learn the transferable features across two domains by imposing the large identical norm restriction on featurenorm expectations. Unlike these feature adaptation methods, [Zhang et al., 2017] proposed joint geometrical and statistical alignment (JGSA), which projected the source and target data into low-dimensional subspaces with two coupled map functions. In the subspace, both the geometrical and statistical shift across two domains were reduced.

3 Method

3.1 ProblemDefinition

In unsupervised 2D image-based 3D shape retrieval, we have access to the labeled source domain (2D) S = {(xsi , ys i )}ns i=1 where ys i ∈ Y = {1, . . . ,C} and unlabeld target domain (3D) T = {xtj }nt j=1 . S and T are assumed to be different but share the same label distribution. The key problem of this task is to learn a feature function G that projects 2D images and 3D shapes into a common embedding space. In our paper, G denotes all the layers up to fc7 including mutual information estimation operation and F represents the source classifier.

3.2 Cross-Modal Instance Feature Learning

This step mainly consists of two modules, the basic CNN network for visual feature extraction on both 2D and 3D domains and a mutual information maximization module for discriminative feature learning.

Visual Feature Extraction

Given a 3D shape, we can set the preset virtual cameras around it to obtain the multi-view image set V = {vi}Ni=1 (N represents the view number) by the Phong reflection model [Phong, 1975]. Similar to the most related works, we set the view number as 12 and the camera array setting follows [Su et al., 2015]. To extract the 2D image and multi-view features, a shared 2D CNN is employed. The backbone CNN in our work is AlexNet [Krizhevsky et al., 2012] pre-trained on ImageNet. The original AlexNet has 8 layers, which contains 5 convolutional layers (conv1-5) and three full-connected layers (fc6-fc8). Then we can obtain low-level image feature map Is ∈ RW×H×C and multi-view embedding tensor It = [i1t , i2t , . . . , iNt ]T ∈ RN×W×H×C from the output of conv5, the high-level image embedding Ks ∈ RD and multiview image embedding matrix Kt = [k1 t , k2 t , . . . , kN t ]T ∈ RN×D from the output of fc7. Note that W,H,C,D are 6, 6, 256, 256 in our experiment. Finally, we impose the maxpooling operation on Kt across different views to obtain the final 3D shape descriptor K′t ∈ RD for retrieval.

Mutual InformationMaximization

The traditional deep learning works pay more attention to the input features and the associated labels, in which the features focus on preserving the commonality of categories. However, the common-class features are not suitable for 3D shape retrieval task. In other words, the intra-class distance should be smaller while the inter-class distance should be big enough in two domains. To meet these requirements, we maximize the mutual information (MI) between the low-level convolution features (Is, It) and the high-level embeddings (Ks,Kt).

Based on [Belghazi et al., 2018], the MI estimation can be computed as the Kullback-Leibler divergence between the joint PMN, and the product of the marginals PM ⊗ PN, i.e. I (M;N) = DKL (PMN‖PM ⊗PN) ,M ∈ {Is, It},N ∈ {Ks, Kt}. However, the unbounded upper limit of DKL may result in the infinite result. To address this problem, [Belghazi et al., 2018; Hjelm et al., 2019] propose to represent the KLdivergence with Donsker-Varadhan representation [Donsker and Varadhan, 1975] to obtain the lower-bound. Specifically, the MI estimator can be rewritten as:

where T can be implemented as a discriminator function with parameter θk in deep neural network. Therefore, Iθ (M;N) can be computed by:

During the training procedure, the MI estimator implemented by a discriminator can approximate the mutual information with arbitrary accuracy gradually by the gradient back propagation. In our architecture, the input of MI estimator is the combination of the low-level features Is/It and high-level features Ks/Kt. "Fake samples" are constructed by combining the same low-level features with high-level features obtained from another image/shape. Note that the 2D and 3D domains share an identical MI estimator. Then the discriminator is trained to distinguish the "real or fake" samples, and we can calculate the binary cross-entropy loss LMI, which can be regarded as the estimation of MI.

3.3 Hierarchical Instance Feature Alignment

This step mainly consists of two modules, global domain statistics alignment and local semantic information alignment for reducing the domain shift.

Global Domain Statistics Alignment

Although we can obtain more discriminative feature for individual instance (2D image or 3D shape) with the help of MI estimation, the gap between two domains has not been reduced. As shown in Figure. 2, due to the large global domain and local semantic variations, the overlap of the feature distributions is small. Inspired by the Misaligned-FeatureNormHypothesis [Xu et al., 2019] that the domain shift is decided by the misaligned feature-norm expectations and consequently the equal value of feature-norm expectations in two domains can reduce the domain shift, we propose to enforce the global domain statistics alignment between the 2D and 3D domains by the Maximum Mean Feature Norm Discrepancy (MMFND),

where H denotes the combination of all the potential feature extraction functions composited by the L2-norm, i.e., h(x) = (‖ · ‖2 ◦G) (x). Intuitively, the function class H without any restriction may result in the great deviation of the upper bound from zero. To avoid it, we impose the identical large norm restriction R on the feature-norm expectations on both 2D and 3D domains. Consequently, the domain gap measured by MMFND will gradually vanish to zero and we can obtain domain-invariant features for retrieval. Thus, the global domain confusion loss LG can be written as:

where Ld(.) represents L2-norm penalty. We empirically set R as 25 by following [Xu et al., 2019] since the lower value is prone to achieve lower accuracy on target domain while a sufficiently large R may lead to the gradient explosion.

Local Semantic Information Alignment

As shown in Figure. 2, the global domain statistics alignment does not imply a local semantic class-to-class alignment. For example, the visual feature of 3D vase may be mapped nearby the embedding of 2D cup while satisfying the global domain statistics alignment since the unlabeled 3D domain adds the semantic ambiguity. Conversely, the 2D domain can preserve the semantic consistency by the classification loss,

where Lc represents the classification cross-entropy loss. Thus, it is necessary to pursue the local semantic information alignment under the absence of true 3D labels. We solve this problem by minimizing the distance between two centroids of the same class from different domains. Unfortunately, we have no access to label information from 3D domain. To circumvent the impossibility of local semantic alignment, we assign pseudo labels to 3D samples with the classifier F. Finally, the local semantic information alignment can be narrowed by minimizing,

where LL is the local semantic loss, K denotes the class number and C k S , C k T are the tth class feature centroid on 2D and 3D domains respectively. We can obtain 2K centroids in total. Obviously, there must be some false pseudo-labeled samples and they may have a negative effect on adaptation. Fortunately, when computing the centroids, all pseudo-labels (false or true) samples are used together and the negative effect brought by fake samples can be neutralized by true pseudolabeld samples to some extent.

The total objective loss can be written as:

where λ, β and γ are hyper-parameters to trade off different loss functions. We set λ, β as 0.05, 1 and γ = 2 1+exp(-10·p) - 1, respectively, where p denotes the training progress varying from 0 to 1. This setting can further reduce the negative effect caused by pseudo-labeled 3D shapes since the pseudo label confidence is low in the early training process. As the discrepancy between the two domains decreases, the pseudo label confidence will increase and consequently the local semantic loss will play a more important role in enforcing the fine-grained class-to-class alignment.

4 Experimental Settings

4.1 Datasets and Evaluation Criteria

MI3DOR. The MI3DOR dataset [Zhou et al., 2019a] contains 21 categories with 21,000 images and 7,690 3D shapes. There are 10,500 images and 3,842 3D shapes for training, while 10,500 images and 3,848 3D shapes for testing.

MI3DOR-2. TheMI3DOR-2 dataset [Zhou et al., 2019a] is composed of 40 categories with 19,694 images and 3,982 3D shapes. MI3DOR-2 dataset is divided into two parts, 19,294 images/3,182 3D shapes are used as the training set and 400 images/800 3D shapes are used for testing.

Evaluation Criteria. For fair comparisons, we employ the common retrieval evaluation criteria as [Zhou et al., 2019a]. It includes the Nearest Neighbor (NN), First Tier (FT), Second Tier (ST), F measure (F), Discounted Cumulative Gain (DCG) and Average Normalized Retrieval Rank (ANMRR). The value of these criteria ranges from 0 to 1 and the higher value means well performance other than ANMRR.

4.2 ComparedMethods

We mainly compare HIFA with several representative methods: 1) basic deep learning method, AlexNet [Krizhevsky et al., 2012], 2) traditional transfer learning method, MEDA [Wang et al., 2018] and JGSA [Zhang et al., 2017]. 3) deep transfer learning method, JAN [Long et al., 2017], RevGard [Ganin and Lempitsky, 2015] and DLEA [Zhou et al., 2019a]. Since the most mentioned methods (other than DELA) are proposed for 2D visual domain adaptation, the max-pool pooling operation are adopted to fuse multi-view features.

5 Experimental Results

5.1 Comparison with the State-of-the-art Methods

Table. 1 and Table. 2 present the experiment results of HIFA and the comparisons among representative methods on MI3DOR and MI3DOR-2, respectively. It's obvious that HIFA can achieve the best retrieval performance. Specifically, on MI3DOR, HIFA is superior to the comparisons with the gain of 1.83%-83.49%, 10.75%-91.33%, 7.26%-63.75%, 5.59%-52.53%, 9.55%-89.56% with respect to NN, FT, ST, F, DCG, and with the decline of 14.01%-45.73% according to ANMRR. On MI3DOR-2, HIFA is superior to the comparisons with the gain of 3.45%-39.96%, 2.70%-60.56%, 4.26%- 45.49%, 2.00%-60.56%, 0.84%-56.14% with respect to NN, FT, ST, F, DCG, and with the decline of 2.59%-34.34% according to ANMRR. Moreover, we can notice the following observations:

HIFA vs. DLEA. DLEA is the most recent method achieving the best performance for this task since it can jointly realize the feature learning and distribution alignment by the domain discriminator and centroid alignment operation. Compared with DLEA, HIFA achieves the gain of 1.83%/3.45%, 10.75%/2.70%, 7.26%/4.26%, 5.59%/2.00% with respect to NN, FT, ST, F, DCG, and the decline of 14.01%/2.59% with respect to ANMRR on MI3DOR and MI3DOR-2, respectively. This result can demonstrate that the mutual information maximization module can preserve as much visual characteristic as possible of individual instance (2D image and 3D shape) for retrieval. Comparatively, DLEA ignores the instance visual information and only focuses on the commonclass features, which will weaken the retrieval performance.

HIFA vs. others. Compared with other methods, HIFA has two main advantages. First, HIFA can jointly eliminate the global domain and local semantic variations by the hierarchical instance feature alignment module. Comparatively, previous methods either enforce the domain alignment or project the features into the identical subspace while neglecting the semantic consistency in two domains. Second, HIFA can learn the discriminative instance features for retrieval. Comparatively, the other methods only concentrate on narrowing the gap between the 2D and 3D domains while neglecting the significance of feature representation for retrieval.

5.2 Discussion

In this section, we explore the retrieval influence caused by the global domain statistics alignment (-G), local semantic information alignment (-L) and mutual information (MI) maximization modules. Besides, we also visualize the features to further analyze the proposed method.

Effect of Global Domain Statistics Alignment. As shown in Table. 3, HIFA-G can achieve the gain of 62.03%/26.45%, 71.16%/46.76%, 58.21%/36.07%, 33.33%/46.76, 72.17%/41.25% with respect to NN, FT, ST, F, DCG, and the decline of 37.18%/26.07% on ANMRR comparing against AlexNet on MI3DOR and MI3DOR-2,

(a) AlexNet

(b) HIFA-G

respectively. The basic deep learning method AlexNet without feature alignment can not narrow the domain discrepancy caused by the global domain and local semantic variations, and consequently the gap between the 2D and 3D domain is large. Comparatively, HIFA-G imposes the large norm constraint on feature-norm expectations to bridge the significant 2D-to-3D domain gap.

Effect of Local Semantic Information Alignment. As shown in Table. 3, HIFA-GL can achieve the gain of 9.61%/2.75%, 5.45%/4.99%, 1.35%/3.31%, 12.12%/4.99%, 7.24%/3.20% with respect to NN, FT, ST, F, DCG, and the decline of 9.55%/6.02% on ANMRR comparing against HIFAG on MI3DOR and MI3DOR-2, respectively. Global domain statistics alignment can contribute to generate domaininvariant features for this task. However, domain-invariance doesn't mean semantic consistency and the class-level difference still exists. Comparatively, HIFA-GL can narrow the local semantic variations by minimizing the distance of class centroids from two domains.

Effect of Mutual Information Maximization. As shown in Table. 3, HIFA-GL (W/ MI) can achieve the gain of 3.32%/7.73%, 3.00%/4.20%, 2.13%/3.50%, 2.03%/4.20%, 2.07%/4.36%, 9.55%/0.84% with respect to NN, FT, ST, F, DCG, and the decline of 1.70%/5.49% on ANMRR comparing against HIFA-GL on MI3DOR and MI3DOR-2, respectively. It's not enough to achieve the well performance only by enforcing the global and local alignment, since the significant visual discriminative information of individual instance has been neglected. Therefore, HIFA with MI can further improve retrieval performance.

Feature Visualization. As shown in Figure. 3, we visualized the features obtained from different settings on MI3DOR dataset by t-SNE [Maaten and Hinton, 2008]. Note that the red and blue points represent the features from the 2D image and 3D shape domains, respectively. A transferable and discriminative feature mapping should mix up the red and blue points, and meanwhile the points can be easier to recognize its cluster. Other than AlexNet, which does nothing to enforce the alignment across two domains, other approaches can learn the domain-invariant features for this task since the global domain statistics alignment module has been adopted. Besides, the features learned by HIFA-GL and HIFA-GL (W/MI) are more suitable for retrieval task since the identical class across two domains aligns better and the gap between the red and blue points is smaller. Representations learned by HIFA-GL (W/ML) behave better than HIFA-GL and features in different classes from two domains are dispersed relatively instead of mixing up. This finding tells us that the mutual information maximization can further minimize the intra-class discrepancy and maximize the inter-class margin.

6 Conclusion

We propose a novel unsupervised hierarchical instance feature alignment network for 2D image-based 3D shape retrieval. It can jointly realize the cross-modal instance feature learning and hierarchical instance feature alignment. Different from previous methods, which only concentrate on the common-class features learning and global domain alignment, our work first introduce MI maximization to obtain discriminative instance feature and enforce the embedding alignment in both global and local levels. Experimental analysis can demonstrate the superiority of the proposed method.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (61772359, 61872267, 61902277), the grant of Tianjin New Generation Artificial Intelligence Major Program (19ZXZNGX00110, 18ZXZNGX00150), the Open Project Program of the State Key Lab of CAD & CG, Zhejiang University (Grant No. A2005, A2012).