PointLIE: Locally Invertible Embedding for Point Cloud Sampling and Recovery

1 Introduction

Recently, as a fundamental representation of 3D data, point cloud collected by various depth scanners or LiDAR sensors has been applied to diverse domains, such as autonomous driving [Yan et al., 2020a] and cultural heritage reconstruction [Xu et al., 2014]. However, with the increasing capabilities of 3D data acquisition, gigabytes of raw point data can be generated per second, e.g., Velodyne HDL-64E can collect up to 2.2 million points per second. Therefore, large-scale point clouds pose great challenges to the subsequent transmission, storage and interconnection. Thus, the Point Cloud Sampling and Recovery (PCSR) task becomes critical for massive realtime point cloud collection and processing, which aims to sample meaningful points from dense point cloud to compress the scale of the original point cloud while preserving the local topology of discarded points for reconstruction.

∗Corresponding author. † Equal first authorship.

In point cloud research, significant progress has been made in the single-track task for better compressing or upsampling the input point clouds, i.e., compressing point cloud with more pleasure surface approximations [Schwarz et al., 2018] or upsampling sparse point cloud to dense point cloud [Yu et al., 2018; Yifan et al., 2019]. However, bi-directional PCSR remains challenging for several reasons: (1) Traditional methods compress point cloud through gradual sparseness. The global geometric configuration is lost in the intermediate process, making the intermediate results useless and invisible to downstream applications; (2) The relationship between data points requires huge storage. (3) The simple upsampling methods usually produce unsatisfactory recovery results due to the loss of local structure information.

To deal with the above issues, we propose a novel scheme, named Locally Invertible Embedding (PointLIE), to achieve adaptive sampling and faithful reconstruction via an invertible network for efficient storage, intermediate visibility and effective recovery. Inspired by the invertible neural network (INN), we design an INN-based framework to record the inter-points relationships without extra storage. As shown in Fig. 1, in the sampling path, PointLIE adaptively samples viewable and recovery-friendly sub-point clouds with arbitrary scales while preserving the local offsets to the discarded points recursively. Furthermore, it explicitly embeds the above local topology of the discarded points into a latent variable constrained to follow a specified distribution. Therefore, when point cloud recovery is needed, PointLIE could reconstruct a faithful dense point cloud simply by sampling a randomly-drawn latent variable from the above distribution and traversing the inverse path of the network together with the sampled sub-points. In this way, the original point cloud can be better restored without introducing additional storage.

In practice, PointLIE contains several Decomposition Modules and Point Invertible Blocks. The former is adopted to decompose the input point features into sampled point features and offsets to the discarded ones. After this, the number of point features is halved, while the dimension of channels increases exponentially due to higher-order offsets (i.e., subtracting the sampled point features from the discarded ones recursively). By applying an elaborately designed crossconnection architecture, Point Invertible Blocks could further characterize the mutual correlation between the sampled features and their offsets in each sampling scale. Following the aforementioned recursive process, PointLIE could encode all the lost information in each sampling scale into a locally invertible embedding. In the training phase, we force such an invertible embedding to conform to a pre-specified distribution (e.g. isotropic Gaussian) by using a distribution fitting loss. Due to the reversible nature of PointLIE, the recovery process can be conducted by passing through PointLIE inversely as illustrated by the green arrows in Fig. 1.

The main contributions of this paper are three folds. 1) To the best of our knowledge, this is the first work that adopts INN for the PCSR task. A novel PointLIE scheme is proposed to model the sampling and upsampling processes into the same network through bi-directional learning. 2) We propose a Decomposition Module to decouple the input point features into the sampled ones and offsets to their corresponding local neighbors, and Point Invertible Blocks to update them in each sampling scale. Meanwhile, a recursive invertible embedding is proposed to transform the offsets of local neighbors into a latent variable that obeys a specific distribution. 3) Extensive experiments demonstrate that PointLIE outperforms the state-of-the-art point cloud sampling and upsampling methods both quantitatively and qualitatively.

2 RelatedWork

2.1 SamplingMethods for Point Clouds

Traditional sampling methods, such as Farthest point sampling (FPS), have wide applications in various point cloud frameworks [Qi et al., 2017; Wu et al., 2019], since they can sample relatively uniformly distributed points. However, they do not consider the subsequent processing of the sampled points and may result in sub-optimal performance. Recently, there are some alternative sampling methods proposed to better capture the information of point clouds. [Nezhadarya et al., 2020] introduced a critical points layer, which retains the critical points with the most active features to the next network layer. [Yan et al., 2020b] adaptively shifted the sampled points to objects' surface and thus increased the robustness of the network in noisy point clouds. Other methods jointly consider sampling with downstream tasks. For example, [Dovrat et al., 2019; Lang et al., 2020] introduced a task-specific sampling, which can improve the results through training with task-specific loss. However, these methods improve the reconstruction mainly by joining the loss of specific tasks, while the geometric information lost in discarded points during sampling is not considered.

2.2 UpsamplingMethods for Point Clouds

Point cloud upsampling aims to improve the point distribution density and uniformity. [Yu et al., 2018] first proposed the neural network PU-Net, which learns point-wise features by PointNet++ [Qi et al., 2017], expanding the point set in feature space, and reconstructs an upsampled point set from those features. 3PU [Yifan et al., 2019] is a multi-step progressive network, which learns different levels of details in multiple steps. However, due to its progressive nature, it requires a large amount of computation and more data to supervise the intermediate output of the network. Recently, a Point Cloud Generative Adversarial Network (PU-GAN) [Li et al., 2019] is designed to learn the distribution of the upsampled point set through adversarial learning. Upsampling is an illposed problem since a downsampled point set corresponds to multiple plausible dense point clouds. Existing deep-learning based methods directly model this ambiguous task by learning the mapping from a sparse point set to a dense one under the supervision of the ground truth dense point set. However, these methods fail to yield faithful complete reconstruction results, since the valuable information lost in the sampling process is ignored and irreversible.

2.3 Invertible Neural Network

Obtaining the measurable quantities (sampled points) from the given hidden parameters (sampling methods) is referred to as the forward process (i.e., sampling). Correspondingly, the inverse process requires to infer the hidden states of a system from measurements (i.e., reconstruction). The inverse process is often intractable and ill-posed because valuable information is lost in the forward process [Ardizzone et al., 2018]. To fully assess the diversity of possible inverse solutions for a given measurement, invertible neural networks (INNs) are employed to estimate the complete posterior of the parameters conditioned by observation, which is widely employed in both generative models [Dinh et al., 2014; Dinh et al., 2016; Kingma and Dhariwal, 2018; Behrmann et al., 2019; Chen et al., 2019] and classification tasks [Jacobsen et al., 2018]. Unlike traditional deep neural networks, which attempt to directly model the ambiguous problem of inferring the non-unique feasible result, INNs focus on learning the determinate forward process, using latent variables to capture the lost information. Due to the invertibility, the inverse process can be obtained for free by running through the network backwards.

3 Methods

3.1 Task Overview

Given a dense point set Q = {qi}Ni=1 , the goal of point cloud sampling and recovery (PCSR) with scale factor r is to adaptively sample it into a sparse sub-point cloud ˆ P = {ˆ pj }N/r j=1 without any extra preservation, and then reconstruct a dense point cloud ˆQ= {ˆ qi}Ni=1 from ˆ P. To achieve the above goal, PointLIE is proposed as shown in Fig. 1. The forward path transforms the dense point cloud into sampled points ˆ P and a local invertible embedding z containing the geometric information lost during sampling. Due to the reversible nature of PointLIE, the inverse path can reconstruct a faithful dense point cloud for free by running through the PointLIE backwards. The whole process can be formulated as,

where fθ(·) denotes the forward path of our model, and z is the locally invertible embedding generated in the forward path, which is made to follow a specific distribution p(z). Note that here z ∼ p(z) is case-agnostic instead of casespecific (i.e., z ∼ p(z| ˆ P)). Therefore, there is no need to store z after sampling, and we can just randomly draw an embedding z⋆ from the distribution p(z) in the inverse path. ˆ P and z⋆ are used to reconstruct a faithful ˆQ through the inverse process f -1 θ (·).

3.2 Invertible Architecture

To achieve the invertible operations, we firstly construct a Rescale Layer by stacking a Decomposition Module and M Point Invertible Blocks (PI Blocks) detailed in Fig. 2. When dealing with the PCSR with the scale factor r, we stack s Rescale Layers (s=⌊log2 r⌋) to obtain the entire framework of PointLIE as illustrated in Fig. 3.

Decomposition Module

As shown in Fig. 2 (a), during the sampling process, the decomposition module decouples the lost geometric information contained in discarded points from the sampled points. Specifically, for input point features with the shape of (N, d), we first conduct farthest point sampling (FPS) to select N/2 points while the remaining N/2 points are regarded as discarded points. To make the network preserve the information in discarded points, for each sampled point qi ∈ Qs, we find its k nearest neighbors in the discarded points and denote the spatial offsets from qi to its neighbors as Qr. Here we use offsets rather than spatial coordinates of its neighbors since deep neural networks are more capable of learning the residues, and it is also easier to make the residues follow the isotropic Gaussian distribution. The decomposition module outputs two branches of features, i.e., offsets to k nearest features Q0r with shape (N/2, d×k) and sampled point features Q0s with shape (N/2, d).

Point Invertible Block

To further characterize the representation of the two branches during the forward path, we design a point invertible block to update features, inspired by the coupling layer in generative models [Dinh et al., 2014; Dinh et al., 2016; Xiao et al., 2020]. As shown in Fig. 2 (b), each PI block takes two branches as input (i.e., the 1/2 sampled point features Qls and their kNN offsets Qlr ) and generates updated features Ql+1 s and offsets Ql+1 r by Eq. (3) (4),

where l denotes passing through the l-th PI block, and F, G, H are three independent nonlinear transformations. We use several stacked conv1d with the nonlinear activation for F, G, and the dense feature extractor in [Yifan et al., 2019] for H. PI blocks only enhance the representation of sampled features and neighboring offsets gradually, while the shapes of inputs and outputs of each PI block remain unchanged.

Recursive Offset Residue Embedding

Fig. 3 illustrates the overall bi-directional pipeline of PointLIE for the PCSR task. By stacking s rescale layers, where each of them contains a decomposition module and M PI blocks, we construct a hierarchical structure for PCSR with arbitrary scales. For each rescale layer, taking point features with shape (N, d) as input, it will generate sampled features with shape (N/2, d) and feature offsets to k nearest neighbors with shape (N/2, k × d). Then a channel-wise concatenation is conducted to merge the sampled features and their neighboring offsets to generate new point features. These 'higher-order' point features will continue to be input to the next rescale layer. Therefore, the final embedding can be expanded to a series of high-order offsets recursively. QMs and QMr generated by the last Rescale Layer are treated as the adaptively sampled sub-point cloud ˆ P and the embedding z.

Inverse Reconstruction Process

To reconstruct the original dense point set, we use the adaptively sampled point set ˆ P and a randomly-drawn embedding z⋆ ∼ p(z) as two branches of input to the reverse path of PointLIE (i.e., rescale layer s, s-1, ..., 1) as indicated by the gray dashed arrows in Fig. 3. In each rescale layer, they will also flow in a reverse direction (i.e., PI block M, M - 1, ..., 1, decomposition module).

The inverse operations of the PI Blocks and the decomposition module are shown in Fig. 2 of supp. In the reverse path, the (l+1)-th PI block aims to recover the neighboring offsets Qlr and the sampled features Qls fromQl+1 r and Ql+1 s . Considering the inputs Ql+1 r ,Ql+1 s with shapes (N, k × d) and (N, d), the reverse process of Eq. (3)(4) can be expressed as,

After reversely passing through M PI blocks, the output Q0r , Q0s will flow into the decomposition module reversely. In detail, Q0r will be evenly split into k offset matrices {Q(i) r }ki=1 along the channel dimension, where Q(i) r with shape (N, d) represents the offsets to the i-th nearest neighbour for each point in Q0s in the discarded points. Then element-wise addition will be conducted between each Q(i) r and Q0s respectively, obtaining features of recovered discarded points Qd with shape (kN, d). Qd will be concatenated with Q0s in a point-wise manner to form a candidate recovered point set Qc with shape ((k + 1) × N, d), whose first three dimensions in d record the spatial coordinates. To guarantee the uniformity of the reconstructed points, we use FPS to select 2N point features fromQc based on their coordinates.

Analogously, these ×2 reconstructed point features will be evenly split into (k + 1) parts, where the first part and the remaining k parts are taken as the sampled point features QMs and the neighboring offsets QMr respectively. Then, they will be fed into the next reversed rescale layer to conduct another ×2 reconstruction. Supported by Theorem 1, a faithful dense point cloud can be reconstructed progressively. The proof is provided in the supp.

Theorem1. Suppose the generated invertible local embedding z is subject to a latent distribution p(z). In the recovery process, by randomly sampling a z⋆ from p(z) and passing it through the reverse path, the reconstructed dense point cloud ˆQ will necessarily conform to the distribution of the real point cloud p(Q).

3.3 Training Objectives

To improve the reconstruction result, PointLIE models the bidirectional transformation between the dense point cloud Q and the sampled point cloud ˆ P with a latent distribution p(z). Therefore, the total loss contains the following parts.

Sparse Point Sampling Loss

Since the generated sampled point cloud ˆ P is not the subset of Q , we adopt the Earth Mover's distance loss (EMD) [Fan et al., 2017] Lsam to restrict ˆ P to approach P.

where P denotes the ground truth sub-point set uniformly sampled from the original dense point set Q by FPS, and φ : ˆ P → P is a bijective mapping.

Dense Point Reconstruction Loss

To reconstruct finer results, besides using EMD loss to restrict the geometric details of the predicted sub-point set ˆ P, the reconstructed point set ˆQ should also be uniformly distributed on the surface of objects, thus repulsion loss Lrep [Yu et al., 2018] and uniform loss Luni [Li et al., 2019] are used to distribute the recovered point set ˆQ uniformly. So the total loss for reconstruction is formulated as,

Distribution Fitting Loss

Distribution fitting loss is used to encourage the distribution of the generated local embedding f z θ (Q) to approach the latent distribution p(z), which is the sufficient condition for the reconstructed point set ˆQ to follow the real distribution of the original dense point set Q as proved in the Theorem 1. In practice, the cross-entropy loss (CE) is employed to measure the difference between the distributions of the generated embedding f z θ (Q) and p(z). Here p(z) is set as an isotropic Gaussian distribution,

Compound Loss

Overall, we train our PointLIE in an end-to-end manner by minimizing the total loss L as follows,

4 Experiments

4.1 Dataset andMetrics

To fully evaluate the proposed PointLIE, we compared our method with the state-of-the-art methods on PU-147 [Li et al., 2019] dataset, which follows the official split of 120/27 for our training and testing sets. We first used the Poisson disk sampling (PDS) to uniformly sample 8192 points from each original mesh as our ground truth dense point set (GT). Then we used different scale factors (i.e., 4, 8 and 16), and sampling modes (i.e., FPS and learnable sampling methods) to sample sub-point cloud from GT, and compared the reconstruction results with GT. To quantitatively evaluate the performance of different methods, three commonly-used metrics are adopted, i.e., Chamfer distance (CD), Hausdorff distance (HD) and point-to-surface distance (P2F). The lower the metric values are, the better the reconstruction results are.

4.2 Implementation Details

Under the premise of balancing efficiency and effectiveness, we set PI block number M as 8 in the 4× scale task, and M as 4 in the rest 8× and 16× tasks. Furthermore, we set k as 3 to ensure that the information in the discarded points can be sufficiently preserved. More details are shown in the supp.

4.3 Quantitative Results

Point Cloud Sampling and Recovery. In Tab. 1 we compared the results of PCSR with recent state-of-the-art methods: PU-Net [Yu et al., 2018], 3PU [Yifan et al., 2019] and PU-GAN [Li et al., 2019]. To fairly compare with previous methods, we used different sampling modes to sample the sub-point clouds (i.e., FPS and adaptive sampling [Lang et al., 2020]). The upper and lower part of Tab. 1 show the results by using FPS and the learnable sampling respectively. Among all, our PointLIE achieved the best results for most of the evaluation metrics, especially for large scale PCSR tasks (×8 and ×16). Furthermore, we compared with the stateof-the-art adaptive sampling methods (i.e., SampleNet) combined with the most appealing upsampling methods. We used the official codes of SampleNet [Lang et al., 2020] and jointly trained it with the downstream upsampling methods (i.e., PUNet and PU-GAN). The results show that SampleNet cannot effectively improve the reconstruction performance.

Point Cloud Upsampling. Our PointLIE can also be used as a general point cloud upsampling framework by feeding a sparse point cloud into the inverse stream of the trained model. For a fair comparison, we followed the experiment setting of [Li et al., 2019], feeding randomly sampled 2048 or 512 points to predict 8192 points. As shown in Tab. 2, our PointLIE achieved comparable results in all evaluation metrics. Particularly, our results by far exceed all previous methods without adversarial learning (e.g., PU-GAN (-)), and even outperforms the complete PU-GAN in ×16 task. This result confirms that the performance improvement of PU-GAN mainly comes from the discriminator rather than the model architecture itself, while our architecture design can achieve superior upsampling results for both dense and sparse input.

4.4 Qualitative Results

We also compared our qualitative results with PU-Net, 3PU and PU-GAN for PCSR on different scales. Here PU-Net and PU-GAN took the points sampled by SampleNet as inputs. Fig. 4 shows the visual results of PCSR and surface reconstruction by [Kazhdan and Hoppe, 2013]. As shown in Fig. 4, PointLIE can reconstruct more fine-grained details, especially for local geometric shapes (e.g., human hands, bird claws and dragon horns), while other methods tend to produce more noisy and nonuniform point sets, resulting in more artifacts and ambiguities on the reconstructed surfaces.

4.5 Ablation Study

To further demonstrate the effectiveness of our proposed method, we design an ablation study for different training modes and data feeding. In Tab. 3, we first show the result produced without bi-directional learning (only training the inverse process) in the first two rows. These results show that only using the inverse process during training cannot make the model learn the distribution of the reconstructed point cloud. Then, we used the proposed bi-directional training strategy mentioned above, which made a remarkable improvement dealing with randomly or uniformly sampled point clouds. Finally, when we used the sub-point set adaptively sampled by our network, further improvement is achieved.

5 Conclusion

We are the first to adopt the INN for the PCSR task and propose a novel framework PointLIE, which models the sampling and upsampling into the same network through bidirectional learning. By using decomposition modules and point invertible blocks to decouple the discarded points from the sampled points and update the features, our PointLIE can finely restore the original point cloud with a recursive invertible embedding using the reversed operations.

Acknowledgments

The work was supported in part by NSFC-Youth 61902335, by Key Area R&D Program of Guangdong Province with grant No. 2018B030338001, by the National Key R&D Program of China with grant No. 2018YFB1800800, by Shenzhen Outstanding Talents Training Fund, by Guangdong Research Project No. 2017ZT07X152, by Guangdong Regional Joint Fund-Key Projects 2019B1515120039, by the NSFC 61931024 & 81922046, by helix0n biotechnology company Fund and CCF-Tencent Open Fund.