AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detection

1 Introduction

Recent advances in deep learning bring rapid progress in autonomous driving. 3D object detection through LiDAR points plays an essential role in understanding the surroundings of the vehicle. LiDAR points can capture precise 3D spatial information for object detection, however, they often suffer from both the lack of semantic information and sparsity of reflected points, leading to failures under foggy or crowded circumstances. Compared to point clouds, RGB images have better strength in providing semantic and long-distance information. Hence, many approaches explore the data fusion of RGB cameras and LiDAR sensors to improve the performance of 3D object detection.

∗Corresponding author (fzhao956@ustc.edu.cn).

Multi-modal 3D object detectors can be roughly grouped into two categories: decision-level fusion and feature-level fusion. The former detects objects in respective modalities and then ensembles the boxes together in the 3D space [Pang et al., 2020]. Different from the decision-level fusion, the feature-level fusion combines the multi-modal features into a single representation from which the objects are detected. Therefore, the detector can fully utilize features from different modalities at the inference stage. In light of this, more approaches for feature-level fusion have been recently developed. One line of work [Vora et al., 2020] projects each point to the image plane and gets the corresponding image features through bilinear interpolation. Although the feature aggregation is performed delicately at the pixel level, in doing so we will lose the dense patterns in the image domain due to the sparsity of the fusion points, i.e., breaking the semantic consistency in the image features. Another line of work [Chen et al., 2017] uses initial proposals provided by 3D detectors to obtain respective RoI features at different modalities and concatenates them together for feature fusion. It maintains the semantic consistency by conducting instance-level fusion, however, it suffers from the coarse feature aggregation and the absence of 2D information at the initial proposal generation phase.

To take the best of these two types of approaches, we propose an integrated multi-modal feature fusion framework for 3D object detection, named AutoAlign. It not only enables the detector to aggregate cross-modal features in an adaptive way, which proves to be effective in modeling relationships between non-homogenous representations. Meanwhile, it takes advantage of the fine-grained feature aggregation at the pixel level, and also preserves the semantic consistency through instance-wise feature interaction (see Figure 1).

Specifically, to preserve concrete details in the RGB data, we design a Cross-Attention Feature Alignment (CAFA) module, which dynamically attends the pixel-level features from the image and maintains the efficiency by fusing features at a higher 3D level (pillar or voxel). Each voxel feature will query the whole image plane to get a pixel-wise semantic alignment map. Then, CAFA aggregates the image features based on the alignment map and concatenates them together with the original 3D features. In order to facilitate the learning of semantic consistency between point clouds and images, we propose a novel Self-supervised Cross-modal Feature Interaction (SCFI) module. In detail, we first use paired 2D3D proposals, predicted by the detector, to extract regional features in their respective domain. After that, a similarity loss will be exerted between paired regional features in 2D and 3D spaces. By interacting with cross-modal features at the instance level, SCFI strengthens the ability to perceive semantic-related information in CAFA.

Additionally, inspired by multi-task learning, we devise a 2D-3D detection joint training paradigm to regularize the optimization of the image branch. Such a training scheme prevents the overfitting issue of the image backbone and further enhances the performance of 3D detectors.

The main contributions of this work are three-fold:

• We propose a learnable multi-modal feature fusion framework, called AutoAlign, which enhances the fusion process at both pixel level and instance level.
• We present a joint training paradigm for 2D-3D detection to regularize the features extracted from the image branch and improve the detection accuracy.
• Through extensive experiments, we validate the effectiveness of the proposed AutoAlign on various 3D detectors and achieve competitive performance on both KITTI and nuScenes datasets.

2 RelatedWork

2.1 3DObject Detection with Single Modality

3D object detection is often conducted through a single modality of either RGB camera or LiDAR sensor. Camerabased 3D methods take the image as input and output the localization of objects in the space. Since monocular cameras cannot provide depth information, these models need to estimate the depth themselves [Chen et al., 2016]. For example, [Mousavian et al., 2017] first predicts the 2D bounding boxes and then estimates the depth of objects to unfold 2D boxes into 3D. However, monocular 3D detection often fails in predicting depth information. Therefore, stereo images are utilized to generate dense point clouds for 3D detection [You et al., 2019; Li et al., 2019]. The most widely used sensors for 3D detection are LiDARs, which can be categorized into three categories: voxel, point, and view. Voxelbased techniques discretize points into voxels and aggregate points into them to extract features [Zhou and Tuzel, 2018]. Different from voxel-based approaches, [Shi et al., 2019] directly processes features at the point level, which maintains the original geometrical information provided by raw points, but they are generally computationally expensive. Extracting features from each view is also a popular stream in 3D detection, where points are compressed into bird-eye view [Lang et al., 2019] or range view [Fan et al., 2021] for instance prediction.

2.2 3DObject Detection withMulti-modalities

Recently, multi-modal fusion for object detection attracts numerous attentions. For example, [Qi et al., 2018] predicts boxes in 2D domain and further refines them in the 3D space. [Ku et al., 2018] and [Chen et al., 2017] attempt to perform RoI-wise fusion. In order to get more smooth BEV maps, [Yoo et al., 2020] proposes to learn an auto-calibrated projection for different modalities. However, it suffers from the feature blurring. Other methods [Sindagi et al., 2019; Liang et al., 2018] fuse features in a point-wise manner. For instance, [Vora et al., 2020] paints the 2D semantic predictions on 3D points using camera projection matrix and then performs 3D object detection. [Huang et al., 2020] designs a novel L1-Fusion module for fine-grained fusion.

3 Method

In this section, we describe the proposed AutoAlign in detail. An overview of our approach is presented in Figure 2.

3.1 Pixel-level Feature Aggregation

Previous works mainly utilize the camera projection matrix to align image and point features in a deterministic manner. This kind of approaches are effective but may bring in two potential problems: 1) the point cannot get a broader view on the image data and 2) only positional consistency is maintained while ignoring semantic correlation. Therefore, we devise the Cross-Attention Feature Alignment (CAFA) module to adaptively align the features between non-homogenous representations. Instead of adopting a one-to-one matching pattern, the CAFA module enables each voxel to perceive the whole image and dynamically attend pixel-wise 2D features based on the learnable alignment maps.

As shown in Figure 2, our method uses ResNet-50 as the backbone to extract global feature maps from given images. As a result, an input image with a size of H×W will produce the feature map with the spatial dimension of H/32×W/32. The feature map extracted from the image backbone is denoted as Z ∈ Rh×w×c , where h,w, c are the height, width, and channel of the global feature map, respectively. An 1×1 convolution is added to reduce the feature dimension, creating a new feature map F ∈ Rh×w×d . After that, we flatten the spatial dimensions of F into one dimension, leading to a hw × d feature vector. In our cross-attention mechanism, given the feature map F = {f1, f2, ..., fhw} (fi indicates the image feature of the ith spatial position) and voxel features P = {p1, p2, ..., pJ} (pj indicates each non-empty voxel feature) extracted from raw point clouds, keys and values are generated from F and queries are produced by P . Formally,

(1) where WQ ∈ Rd×dk ,WK ∈ Rd×dk , and WV ∈ Rd×dv are linear projections. For the j th query Qj , the attention weights are calculated based on the dot-product similarity between the cross-modal query and the key:

where √dk is a scaling factor. The output of the crossattention mechanism is defined as the weighted sum over all values according to the attention weights:

The normalized attention weight si,j models the interests between different spatial pixel fi and voxel pj, which is the align map shown in Figure 2. The weighted sum of the values can aggregate fine-grained spatial pixels to update pj, which enriches the point features with 2D information in a global view manner. Like the transformer architecture, we use the feed-forward network to produce the final RGB-aware point features as:

where FFN(·) is a simple neural network using one fullyconnected (FC) layer [Vaswani et al., 2017].

3.2 Instance-level Feature Interaction

The CAFA is a fine-grained paradigm in aggregating image features. However, it fails in capturing instance-level information. On the contrary, RoI-wise feature fusion maintains the integrity of the object while suffering from its coarse feature aggregation and the absence of 2D information during the proposal generation phase.

To bridge the gap between pixel-level and instance-level fusions, we introduce the Self-supervised Cross-modal Feature Interaction (SCFI) module to guide the learning of the CAFA. It directly utilizes the final predictions of 3D detector as proposals, which leverages both image and point features for accurate proposal generation. Moreover, instead of concatenating cross-modal features together for further box refinement, we conduct the similarity constraint between paired cross-modal features, as a manner of instance-level guidance for feature alignment.

Given 2D feature map F and the corresponding 3D voxelized features P, we randomly sample N regional 3D detection boxes, denoted as B3D = (B3D 1 ,B3D 2 , ...,B3D N ), and then project them into the 2D plane using the camera projection matrix, resulting in a set of paired 2D boxes B2D = (B2D 1 ,B2D 2 , ...,B2D N ). Once obtaining the paired boxes, we adopt 2DRoIAlign [He et al., 2017] and 3DRoIPooling [Shi et al., 2020] in both 2D and 3D feature spaces to obtain respective RoI features R3D and R2D , where each R3D i and R2D i are given by:

For each paired 2D and 3D RoI features, we perform selfsupervised cross-modal feature interaction on C5 from the image branch and the feature after voxelization from the point branch. Both of them are fed into a projection head h, transforming the output of one modality to match another modality. Similar to [Chen and He, 2021], a prediction head f with two FC layers is introduced. Denoting the two output vectors as p1 = f(h(R3D )) and q2 = h(R2D ), we minimize the feature distance D(p1, q2) with negative cosine similarity loss, as illustrated in Figure 3. To pull the two modal representations closer, we define a symmetry loss as:

Besides, the stop-gradient strategy is also adopted for the branch without prediction heads, which can be represented by D(m1, stopgrad(v1)). Hence, the interaction loss is implemented as:

3.3 Joint Training for 2D-3DDetection

Despite the effectiveness of multi-task learning, there is little work discussing the joint-detection for both image and point domains. In most previous methods, the image backbone is directly initialized with the pre-trained weights from other external datasets. During the training phase, the only supervision is the 3D detection loss, propagated from the point branch. Considering the large number of parameters in the image backbone, the 2D branch is more likely to get overfitting with implicit supervision. To regularize the representation extracted from the image, we extend the image branch into Faster R-CNN and supervise it with 2D detection loss, where the total loss L is designed as:

4 Experiments

4.1 Implementation Details

To validate the effectiveness of our AutoAlign, we select PointPillar [Lang et al., 2019], SECOND [Yan et al., 2018], and CenterPoint [Yin et al., 2021a] as representative methods for our experiments. For the image branch, Faster RCNN [Ren et al., 2015] with ResNet50 is adopted as the 2D detector. The hidden units of cross-attention alignment module are set to 128 and the output sizes of 2DRoIAlign and 3DRoIPooling are both set to 4. The MLP units of the projector and predictor of the self-supervised cross-modal module are 2048 and the hidden unit number is 512. Our 2D-3D joint training framework is optimized in an end-to-end manner with hybrid optimizers, where the 3D branch is optimized with AdamW and the 2D branch is optimized with SGD. We use MMDetection3D [Contributors, 2020] as our codebase, and apply the default settings if not specified.

4.2 Results on KITTI Dataset

In this section, we evaluate our framework on the KITTI dataset and report the average precision (AP40). We implement AutoAlign on two representative 3D object detectors: PointPillar (pillar-based) and SECOND (voxel-based). The 3D mAP performance is reported in Table 1. Overall, our AutoAlign significantly improves PointPillar and SECOND by 3.0 and 2.3 mAP under 3D moderate evaluation protocol, which validates the effectiveness of the proposed method. When observing the results in detail, we find that the APs of Pedestrian and Cyclist are promoted most (3.0 and 3.5 mAP on AP3D moderate, respectively). We infer the reason that cars often hold more points, while objects like pedestrians and cyclists are mostly short of reflection, which makes them harder to be detected in the 3D space. Therefore, AutoAlign benefits from the RGB data that are naturally dense and rich in semantic and texture information.

4.3 Results on NuScenes Dataset

We also conduct experiments on the much larger nuScenes dataset with current state-of-the-art 3D detector CenterPoint to further validate the effectiveness of AutoAlign. As shown in Table 3, AutoAlign achieves 66.6 mAP and 71.1 NDS on the nuScenes validation set, outperforming the strong CenterPoint baseline by 7.0 mAP and 6.5 NDS. It also surpasses the recently developed multi-modal 3D detector MVP [Yin et al., 2021b] by 1.1 NDS under the same single-stage settings. Besides, due to its simplicity and joint-training paradigm, it does not require any sophisticated virtual point generation or image feature pre-fetching, which is much more suitable for real-world applications. We also report detailed results on each object category as well as the performance on the testing leaderboard in the supplementary material.

4.4 Ablation Studies

To understand how each module in AutoAlign promotes the detection accuracy, we test each component on the baseline detector SECOND and report its AP performance on the KITTI validation dataset in Table 2.

When cross-attention feature alignment is applied, the accuracy is raised by 0.5 mAP and the improvements are found on objects of all difficulty levels. This result validates the importance of preserving the high resolution of image information when aggregating cross-modal features.

Then, we add the SCFI module which brings in a 1.2-mAP enhancement, namely the overall moderate AP3D improves from 68.5 to 69.7, suggesting that feature interaction plays a pivotal role in our fusion framework. It exerts instancelevel supervision on the feature alignment, which hints at how to aggregate semantically paired features across nonhomogenous representations.

When 2D joint training is added, the accuracy is boosted by another 0.6 mAP and the APhard is raised by 1.0 mAP. Such a large improvement benefits from two aspects: 1) the joint training paradigm regularizes the optimization of image backbone and 2) joint optimization reduces the training gap between 2D and 3D models and maintains the feature consistency during the cross-modal feature fusion process.

4.5 Discussions

Investigating the Best Cross-modal Query Strategy.

In this part, we compare various strategies for cross-modal feature query. Firstly, we choose the widely adopted fusion strategy, i.e., projecting points to the image plane through the camera projection matrix and utilizing point-wise bilinearinterpolation to obtain the aligned 2D image. Secondly, we test the non-local block proposed in [Wang et al., 2018], where all image features are taken into account but only interested positions with high attention scores will be aggregated for cross-modal fusion. Finally, we adopt a more general form, which is similar to the self-attention module in [Vaswani et al., 2017], but we extend it from the same modalities to non-homogenous representations. Following the common design of self-attention, we explore the performance of single-head cross-attention module and the multihead one. The detailed results are listed in Table 4. When using point-based projection, the improvement is limited, since the points are unable to get continuous image features. However, when replacing the point-based projection with nonlocal block, the performance is still unsatisfying. Compared to vanilla non-local block, the performance of cross-attention is more competitive, probably due to the adoption of dropout strategy and feature normalization. Considering the computational cost and efficiency, we finally take single-head crossattention as our query strategy.

Seeking the Suitable Feature Source for Self-supervised Feature Interaction.

Feature interaction is a core component since it intensifies the semantic consistency of the CAFA module with instancelevel guidance. Hence, how to select suitable feature source for self-supervised learning is non-trivial. After carefully examining the selection of point feature and image feature sources, we take the image features directly from ResNet backbone (i.e., C5) and after FPN (i.e., P5) as candidates. For the point branch, we select the features before point backbone and after backbone. As shown in Table 5, using C5 as the image feature is better than P5. We infer the reason that P5 is directly for 2D detection and therefore limits the generalization ability for cross-modal feature fusion, while C5 is more flexible for both 2D detection and non-homogenous self-supervised learning. When choosing point features after backbone, we observe a quick convergence of the similarity loss, but the result is unsatisfying. This may stem from the too much flexibility of the 3D branch, i.e., the point backbone provides possibility of complex transformation for point features, which eases the optimization of loss but weakens the instance-level guidance of semantic consistency by our proposed self-supervised feature interaction. On the contrary, although using the features before backbone slows down the convergence, the model is implicitly supervised by mutual interaction and gradually learns how to align cross-modal features in the CAFA module.

Optimal Loss for Self-supervised Cross-modal Learning.

Since most self-supervised learning approaches are based on homogenous representations, exploring the optimal selfsupervised loss for cross-modality is necessary. We compare four different prototypes and report the results in Table 6. We adopt the classical version of contrastive loss, where both positive and negative pairs are considered. Note that features located at the same position in the 3D space and 2D plane are considered as positive pairs while the rest ones are negative pairs. The selections of NCE loss and its variant InfoNCE do not provide remarkable enhancement. However, when utilizing positive pairs only for feature interaction, we observe significant improvements. We infer the reason that the points hold less identity information compared to images. When supervising two similar instances with negative pair loss, it may deteriorate the 3D feature representation if the shapes of the instances are similar to each other. Therefore, we choose the negative cosine similarity loss with positive pairs for our feature interaction module.

4.6 Visualization and Analysis

In addition to the detection results on various datasets, we also provide a more straightforward visualization on the learned alignment map, given different 3D query voxel features, as shown in Figure 4. To better illustrate the mutual influence between the CAFA and SCFI modules, we compare the query attention map with and without SCFI. It can be clearly concluded that CAFA fails to yield a meaningful alignment map on 2D images without feature interaction. Conversely, when armed with SCFI, the CAFA module can successfully provide a positionally and semantically reasonable feature alignment map.

5 Conclusion

In this work, we develop AutoAlign, a learnable multi-modal feature fusion method for 3D object detection. The proposed Cross-Attention Feature Alignment module enables each voxel feature to aggregate image information in a finegrained manner. Furthermore, a novel Self-supervised Crossmodal Feature Interaction module is designed to enhance the semantic consistency during assignment for the CAFA module. Comprehensive experimental results have demonstrated that AutoAlign significantly improves various 3D detectors on the KITTI and nuScenes datasets.

Acknowledgments

This work was supported by the USTC-NIO Joint Research Funds KD2111180313. We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.