3DViewGraph: Learning Global Features for 3D Shapes from A Graph of Unordered Views with Attention

1 Introduction

Global features of 3D shapes can be learned from raw 3D representations, such as meshes, voxels, and point clouds. As an alternative, a number of works in 3D shape analysis employed multiple views [Su and others, 2015; Han et al., 2019b] as raw 3D representation, exploiting the advantage that multiple views can facilitate understanding of both manifold and nonmanifold 3D shapes via computer vision techniques. Therefore, effectively and efficiently aggregating comprehensive information over multiple views, is critical for the discriminability of learned features, especially in deep learning models.

∗Corresponding Author

Pooling was designed as a procedure for information abstraction in deep learning models. In order to describe a 3D shape by considering features from multiple views, view aggregation is usually performed by max or mean pooling, where pooling only employs the max or mean value of each dimension across all view features [Su and others, 2015]. Although pooling is able to eliminate the rotation effect of 3D shapes, both the content information within views and the spatial relationship among views cannot be fully preserved. As a consequence, this limits the discriminability of learned features. In this work, we address the challenge to learn 3D features in a deep learning model by more effectively aggregating the content information within individual views, and the spatial relationship among multiple unordered views.

To tackle this issue, we propose a novel deep learning model called 3D View Graph (3DViewGraph), which learns 3D global features from multiple unordered views. By taking multiple views around a 3D shape on a unit sphere, we represent the shape as a view graph formed by the views, where each view denotes a node, and the nodes are fully connected with each other by edges. 3DViewGraph learns highly discriminative global 3D shape features by simultaneously encoding both the content information within the view nodes, and the spatial relationship among the view nodes.

1. We propose a novel deep learning model called 3DViewGraph for 3D global feature learning by effectively aggregating multiple unordered views. It not only encodes the content information within all views, but also preserves the spatial relationship among the views.
2. ii) We propose an approach to learn a low-dimensional latent semantic embedding of the views by directly capturing the similarities between each view and a set of latent semantic patterns. As an advantage, 3DViewGraph

• avoids mining the latent semantic patterns across the whole training set explicitly.
• iii) We perform view aggregation by integrating a novel spatial pattern correlation, which encodes the content information and the spatial relationship in each pair of views.
• iv) We propose a novel attention mechanism to increase the discriminability of learned features by highlighting the unordered view nodes with distinctive characteristics and depressing the ones with appearance ambiguities.

2 RelatedWork

Deep learning models have made a big progress on learning 3D shape features from different raw representations, such as meshes [Han and others, 2018], voxels [Wu and others, 2016], point clouds [Qi and others, 2017] and views [Su and others, 2015]. Because of page limit, we focus on reviewing view-based deep learning models to highlight the novelty of our view aggregation.

2.1 View-basedMethods

View-based methods represent a 3D shape as a set of rendered views [Kanezaki et al., 2018] or panorama views [Sfikas and others, 2017]. Besides direct set-to-set comparison [Bai and others, 2017], pooling is the widely used way of aggregating multiple views in deep learning models [Su and others, 2015]. In addition to global feature learning, pooling can also be used to learn local features [Huang et al., 2017; Yu et al., 2018] for segmentation or correspondence by aggregating local patches.

Although pooling can aggregate views on the fly in the models, it can not encode all the content information within views and the spatial relationship among views. Thus, the strategies of concatenation [Savva and others, 2016], view pair weighting [Johns et al., 2016], cluster specified pooling [Wang and others, 2017], RNN [Han and others, 2019], were employed to resolve this issue. However, these methods can not learn from unordered views or fully capture the spatial information among unordered views.

To resolve the aforementioned issues, 3DViewGraph aggregates unordered views more effectively by simultaneously encoding their content information and spatial relationship.

2.2 Graph-basedMethods

To handle the irregular structure of graphs, various methods have been proposed [Hamilton and others, 2017]. Although we formulate the multiple views from a 3D shape as a view graph, existing methods proposed for graphs cannot be directly used for learning the 3D feature in our scenario. The reasons are two-fold. First, these methods mainly focus on how to locally learn meaningful representation for each node in a graph from its raw attributes rather than globally learning the feature of the whole graph. Second, these methods mainly learns how to process the nodes in a graph with firm order, while the order of views involved in 3DViewGraph are always ambiguous because of the rotation of 3D shapes.

Moreover, some methods have employed graphs to retrieve 3D shapes from multiple views [Anan et al., 2015;

An-An et al., 2016]. Different from these methods, 3DViewGraph employs a more efficient way of view aggregation in deep learning models, which makes the learned features useful for both classification and retrieval.

3 3DViewGraph

3.1 Overview

Fig. 1 shows an overview of 3DViewGraph, where the global feature F i ∈ R1×F of a 3D shape mi is learned from its corresponding view graph Gi . Here, mi is the i-th shape in a training set of M 3D shapes, where i ∈ [1,M]. Based on the F-dimensional feature F i , 3DViewGraph classifies mi into one of L shape classes according to the probability P i = [P i (li = 1|F i ), ..., P i (li = a|F i ), ..., P i (li = L|F i )], which is provided by a final softmax classifier (Fig. 1(f)), where li is the class label of mi .

We first take a set of unordered views vi = {vij |j ∈ [1, V ]} on a unit sphere centered at mi , as shown in Fig. 1(a). Here, we use "unordered views" to indicate that the views cannot be organized in a sequential way. The views vij are regarded as view nodes Dij (briefly shown by symbols) of an undirected graph Gi , where each Dij is fully connected with other view nodes Dij ′ by edges Ei j,j ′ , such that Gi = ({Dij }, {Ei j,j ′ }).

Next, we extract low-level features f i j of each view vij using a fine-tuned VGG19 network [Simonyan and Zisserman, 2014], as shown in Fig. 1(b), where f i j ∈ R1×4096 is extracted from the last fully connected layer. To obtain lower-dimensional, semantically more meaningful view features, we subsequently learn a latent semantic mapping Φ (Fig. 1(c)) to project a low-level view feature f i j into its latent semantic embedding dij .

To resolve the effect of rotation, 3DViewGraph encodes the content and spatial information of Gi by exhaustively computing our novel spatial pattern correlation between each pair of view nodes. As illustrated in Fig. 1(d), we compute the pattern correlation cij,j ′ between Dij and each other node Dij ′ , and we weight it with their spatial similarity sij,j ′ . In addition, for each node Dij , we compute its cumulative correlation Ci j to summarize all spatial pattern correlations as the characteristics of the 3D shape from the j-th view node Dij .

Finally, we obtain the global feature F i of shape mi by integrating all cumulative correlations Ci j with our novel attention weights αi , as shown in Fig. 1(e) and (f). αi aims to highlight the view nodes with distinctive characteristics while depressing the ones with appearance ambiguity.

3.2 Latent Semantic Mapping Learning

To learn global features from unordered views, 3DViewGraph encodes the content information within all views and the spatial relationship among views in a pairwise way. 3DViewGraph relies on the intuition that correlations between pairs of views can effectively represent discriminative characteristics of a 3D shape, especially considering the relative spatial position of the views. To implement this intuition, each view should be encoded in terms of a small set of common elements across all views in the training set. Unfortunately, the low-level features f i j are too high dimensional and not suitable as a representation of the views in terms of a set of common elements.

To resolve this issue, 3DViewGraph introduces a latent semantic mapping Φ by learning a kernel function K to directly capture the similarities between V views vij and N latent semantic patterns {ϕn}. Our approach avoids additionally and explicitly mining {ϕn} across the whole training set as the common elements. Φ projects low-level view features f i j into latent semantic space spanned by {ϕn} as latent semantic emdeddings dij . dij represents view nodes Dij with more semantic meaning but lower dimension than f i j . Specifically, predicted by kernel K, the n-th dimension of dij characterizes the similarity between f i j and the n-th semantic pattern ϕn, such that dij = [K(f i j , ϕ1), ..., K(f i j , ϕn), ..., K(f i j , ϕN)] ∈ R1×N . We define the kernel K as

where the similarity K(f i j ,ϕn) is inversely proportional to the distance between f i j and ϕn through exp(), and gets normalized across the similarities between f i j and all ϕn. Parameter β controls the decay of the response with the distance. This equation can be further simplified by cancelling the norm of f i j from the numerator and the denominator as follows,

where in the last step, we substituted 2βϕTn and -β∥ϕn∥22 by ωn and εn, respectively. Here, {ωn}, {εn} and {ϕn} are sets of learnable parameters, in addition, both {ωn} and {εn} depend on {ϕn}. However, to obtain more flexible training by following the viewpoint in [Arandjelovic and others, 2016], we employ two independent sets of {ωn}, {εn}, decoupling {ωn} and {εn} from {ϕn}. This decoupling enables 3DViewGraph to directly predict the similarity between f i j and ϕn by the kernel K without explicitly mining ϕn across all low-level view features in the training set.

Based on the last line in Eq. 2, we implement the latent semantic mapping Φ as a row-wise convolution with each pair of {ωn} and {εn} corresponding to a filter and a row-wise softmax normalization, as shown in Fig. 2.

3.3 Spatial Pattern Correlation

The pattern correlation cij,j ′ aims to encode the content of view nodes Dij and Dij ′ . cij,j ′ makes the semantic patterns that co-occur in both views more prominent while the nonco-occurring ones more subtle. More precisely, we use the latent semantic embeddings dij and dij ′ to compute cij,j ′ as follows,

where cij,j ′ is a N × N dimensional matrix whose entry cij,j ′(n, n′ ) measures the correlation between the semantic pattern ϕn contributing to dij and ϕn′ contributing to dij ′ .

We further enhance the pattern correlation cij,j ′ between the view nodes Dij and Dij ′ by their spatial similarity sij,j ′ , which forms the spatial pattern correlation sij,j ′cij,j ′ .

Fig. 3 visualizes how we compute the spatial similarity sij,j ′ . In Fig. 3(a), we show all edges Ei j,j ′ connecting Dij to all other view nodes Dij ′ in different colors, where Dij is briefly shown by symbols. The length of Ei j,j ′ is measured by the length of the shortest arc connecting the two view nodes Dij and Dij ′ on the unit sphere. Thus, Ei j,j ′ = 2π × 1 × (θ/2π) = θ as illustrated in Fig. 3(b), where θ is the central angle of the arc and the factor 1 corresponds to the radius of the unit sphere. To reduce the high variance of {Ei j,j ′}, we employ Ei j,j ′ = 0.5(1 - cos θ) instead of Ei j,j ′ = θ, which normalizes Ei j,j ′ into the range of [0, 1]. Finally, sij,j ′ is inversely proportional to Ei j,j ′ as follows,

where σ is a parameter to control the decay of the response with the edge length. In Fig. 3(c), we visualize sij,j ′ by mapping the value of sij,j ′ to the width of edges Ei j,j ′ .

To represent the characteristics of 3D shape mi from the jth view node Dij on Gi , we finally introduce the cumulative correlation Ci j , which encodes all spatial pattern correlations starting fromDij as follows,

3.4 Attentioned Correlation Aggregation

Intuitively, more views will provide more information to any deep learning model, which should allow it to produce more discriminative 3D features. However, additional views may also introduce appearance ambiguities that negatively affect the discriminability of learned features, as shown in Fig. 4.

To resolve this issue, 3DViewGraph employs a novel attention mechanism in the aggregation of the 3D shape characteristics from all unordered view nodes of a shape, as illustrated in Fig. 1(e). 3DViewGraph learns attention weights αi = {αij |j ∈ [1, V ]} for all view nodes Dij on Gi , where αij would be a large value (the second row in Fig. 4) if the view vij has distinctive characteristics, while αij would be a small value (the first row in Fig. 4) if vij exhibits appearance ambiguity with views from other shapes. Note that ∑Vj=1 αij = 1.

Our novel attention mechanism evaluates how distinctive each view is to the views that 3DViewGraph has processed. To comprehensively represent the characteristics of the views that 3DViewGraph has processed, the attention mechanism employs the fully connected weights WF in the final softmax classifier which accumulates the information of all views, as shown in Fig. 1(f). The attention mechanism projects the characteristics Ci j of 3D shape mi from the j-th view node Dij and the characteristics WF of the views that 3DViewGraph has processed into a common space to calculate the distinctiveness of view vij , as defined below,

where WC, ωC, ωF , b and ω are learnable parameters in the attention mechanism, WF ∈ RL×F , where F is the dimension of the learned global feature F i , and L is the number of shape classes. With WC ∈ RL×V and ωC ∈ RV×1 , Ci j is projected into a L × 1 dimensional space, where b ∈ RL×1 is a bias in that space. In addition, WF is projected into the same space by ωF ∈ RK×1 to compute the similarities between Ci j andWF along all L dimensions. Subsequently, the attentionweight αij is calculated by comprehensively summarizing all similarities along all the L dimensions with a linear mapping ω ∈ R1×L. Finally, the αij in αi for all views of i-th shape are normalized by softmax normalization.

Based on αi , the characteristics Ci j of 3D shape mi from all view nodes are aggregated with weighting αi into attentioned correlation aggregation Ci , as defined below,

where Ci represents 3D shape mi as a N × N matrix, as shown in Fig. 1(e). Finally, the global feature F i of 3D shape mi is learned by a fully connected layer with attentioned correlation aggregation Ci as input, as shown in Fig. 1(f), where the fully connected layer is followed by a sigmoid function.

Using F i , the final softmax classifier computes the probabilities P i to classify the 3D shape mi into one of L shape classes as

where WF ∈ RL×K and bF ∈ RK×1 are learnable parameters for the computation of P i . WF is used to represent all the characteristics of views that 3DViewGraph has processed, as employed to calculate αi in Eq. 6.

3.5 Learning Inference

The parameters involved in 3DViewGraph are optimized by minimizing the log-likelihood O over M 3D shapes in the training set, where Qi is the truth label,

The parameter optimization is conducted by back propagation of classification errors of 3D shapes. Noteworthy, WF is updated by two elements with the learning rate ε as follows,

The advantage of Eq. (10) is that WF can be learned more flexibly for optimization convergence. WF also enables αi to simultaneously observe the characteristics of shape mi from each view node Dij and take all views that have been processed from different shapes as reference.

4 Results and Analysis

We evaluate 3DViewGraph by comparing it with the stateof-the-art methods in shape classification and retrieval under ModelNet40 [Wu and others, 2015], ModelNet10 and ShapeNetCore55 [Savva and others, 2017]. We also show ablation studies to justify the effectiveness of novel elements.

4.1 Parameters

We first explore how the important parameters F, N and σ affect the performance of 3DViewGraph under ModelNet40. The comparison in Table. 1, 2, and 3 shows that their effects are slight in a proper range.

4.2 Classification

As compared under ModelNet in Table 4, 3DViewGraph outperforms all the other methods under the same condition 1 . In addition, we show the single view classification accuracy in VGG fine-tuning ("VGG(ModelNet)"). To highlight the contribution of VGG fine-tuning, spatial similarity, and attention, we remove fine-tuning ("Ours(No finetune)") or set all spatial similarity ("Ours(No spatiality)") and attention ("Ours(No attention)") to 1. The degenerated results indicate these elements are important for 3DViewGraph to achieve high accuracy. Similar phenomena is observed when we justify the effect of Ci j andWF in Eq. 6 by setting them to 1 ("Ours(No attention-)"), respectively. We also justify the latent semantic embedding and spatial pattern correlation by replacing them by single view features ("Ours(No latent)") and summation ("Ours(No correlation)"), the degenerated results also show that they are important elements. Finally, we compare our proposed view aggregation with mean ("Ours(MeanPool)") and max pooling ("Ours(MaxPool)") by directly pooling all single view features together. Due to the loss of content information in each view and spatial information among multiple views, pooling performs worse.

1 We use the same modality of views from the same camera system for the comparison, where the results of RotationNet are from Fig.4 (d) and (e) in https://arxiv.org/pdf/1603.06208.pdf. Moreover, the benchmarks are with the standard training and test split.

3DViewGraph also achieves the best under the more challenging benchmark ShapeNetCore55, based on the fine-tuned VGG ("VGG(ShapeNetCore55)"), as shown in Table 5. We also find that different parameters do not significantly affect the performance, such as N and σ.

4.3 Attention Visualization

We visualize the attention learned by 3DViewGraph under ModelNet40, which demonstrates how 3DViewGraph understands 3D shapes by analyzing views on a view graph. In Fig. 5, attention weights αi on view nodes Dij of Gi are visualized as a vector which is represented by scattered black nodes, where the corresponding views are also shown nearby, such as the views of a toilet in Fig. 5(a), a table in Fig. 5(b) and a cone in Fig. 5(c). The coordinates of black nodes along the y-axis indicate how much attention 3DViewGraph pays to the corresponding view nodes. In addition, the views that is paid the most and least attention to are highlighted by the red upward and blue downward arrow, respectively.

Fig. 5 demonstrates that 3DViewGraph is able to understand each view, since the view with the most ambiguous appearance in a view graph is depressed while the view with the most distinctive appearance is highlighted. For example, the most ambiguous views of toilet, table and cone merely show some basic shapes that provide little useful information for classification, such as the rectangles of the toilet and table, and the circle of the cone. In contrast, the most distinctive views of toilet, table and cone exhibit more unique and distinctive characteristics.

4.4 Retrieval

We evaluate the retrieval performance of 3DViewGraph under ModelNet in Table 7. We outperform the state-of-the-art methods, where the retrieval range is also shown. We further detail the precision and recall curves of these results in Fig. 7. In addition, 3DViewGraph also achieve the best results under ShapeNetCore55 in Table 6. We compare 10 state-of-the-art methods under testing set in the SHREC2017 retrieval contest [Savva and others, 2017] and Taco [Cohen et al., 2018], where we summarize all the 10 methods ("All") by presenting the best result of each metric due to page limit. Finally, we demonstrate that 3DViewGraph is also superior to other graph-based multi-view learning methods [Anan et al., 2015; An-An et al., 2016] under Princeton Shape Benchmark (PSB) in Fig. 6.

5 Conclusion

In view-based deep learning models for 3D shape analysis, view aggregation via widely used pooling, leads to information loss about content and spatial relationship of views. We propose 3DViewGraph to address this issue for 3D global feature learning by more effectively aggregating unordered views with attention. By organizing unordered views taken around a 3D shape into a view graph, 3DViewGraph learns global features of the 3D shape by simultaneously encoding both the content information within view nodes and the spatial relationship among the view nodes. Through a novel latent semantic mapping, low-level view features are projected into a meaningful, lower-dimensional latent semantic embedding using a learned kernel function, which directly captures the similarities between low-level view features and latent semantic patterns. The latent semantic mapping successfully facilitates 3DViewGraph to encode the content information and the spatial relationship in each pair of view nodes using a novel spatial pattern correlation. Further, our novel attention mechanism effectively increases the discriminability of learned features by efficiently highlighting the unordered view nodes with distinctive characteristics and depressing the ones with appearance ambiguity. Our results in classification and retrieval under three large-scale benchmarks show that 3DViewGraph can learn better global features than the stateof-the-art methods due to its more effective view aggregation.

Acknowledgments

This work was supported by National Key R&D Program of China (2018YFB0505400), NSF (1813583), University of Macau (MYRG2018-00138-FST), and FDCT (273/2017/A). We thank all anonymous reviewers for their constructive comments.