DCDet: Dynamic Cross-based 3D Object Detector

1 Introduction

3D object detection plays a crucial role in enabling unmanned vehicles to perceive and understand their surroundings, which is fundamental for ensuring safe driving. Label assignment is a key process for training 3D object detectors. The dominant label assignment strategies in 3D object detection are anchor-based [Shi et al., 2020a; Xu et al., 2022; Zheng et al., 2021] and center-based [Ge et al., 2020; Hu et al., 2022; Yin et al., 2021; Wang et al., 2021]. However, both of these label assignment schemes encounter issues that limit the performance of detectors.

The anchor-based label assignment generally encounters an imbalanced problem when assigning positive samples to objects with different scales. It employs the prior knowledge of spatial scale for each category to predefine fixed-size anchors on the grid map. By comparing the intersection over union (IoU) between anchors and ground-truth boxes, positive anchors are determined to classify and regress objects. Consequently, the anchor-based label assignment tends to exhibit an uneven distribution of positive anchors across objects of different sizes. For example, car objects typically have a significantly higher number of positive anchors compared to pedestrian objects. This imbalance poses a challenge during training and leads to slow convergence for small objects. Moreover, the anchor-based label assignment scheme necessitates the recalculation of statistical data distribution for different datasets to obtain optimal anchor sizes. This requirement may reduce the robustness of a trained detector when applied to datasets with distinct data distributions.

∗Corresponding author.

Appendix is presented at https://arxiv.org/abs/2401.07240.

The center-based label assignment scheme often faces challenges in providing adequate positive samples for training. This approach has recently been adopted by various 3D object detectors [Ge et al., 2020; Hu et al., 2022; Yin et al., 2021; Wang et al., 2021]. It focuses solely on object centers as positive samples (similar to positive anchors). As a result, the number of positive samples remains consistent across objects of different scales, solving the issue of imbalanced positive sample distribution encountered in anchorbased label assignment. However, the center-based label assignment overlooks many potential high-quality positive samples, as only one positive sample per object is responsible for regressing object attributes. This leads to an inefficient utilization of training data and sub-optimal network performance.

To simultaneously address the aforementioned challenges, this paper introduces a dynamic cross label assignment (DCLA), which aims to provide balanced and ample highquality positive samples for objects of different scales. Specifically, DCLA dynamically assigns positive samples for each object within a cross-shaped region. The size of this region is determined by a distance parameter, which represents the Manhattan distance from the object's center point. Given the varying scale and potential missing points in point clouds, a dynamic selection strategy is employed to adaptively choose positive samples from the cross-shaped region. As a result, each object is assigned sufficient positive samples, and objects of different scales receive a similar number of positive samples, effectively mitigating the issue of positive sample imbalance.

Moreover, a rotation-weighted IoU (RWIoU) is introduced to accurately regress objects. In the 2D domain, the IoUbased loss [Rezatofighi et al., 2019; Zheng et al., 2020; Zhang et al., 2022a] is confirmed to be better than the Lnorm loss. However, in 3D object detection, the development of the IoU-based loss lags behind its 2D counterpart. This challenge arises due to the increased degrees of freedom in the 3D domain. The proposed RWIoU utilizes the idea of rotation weighting, thus elegantly integrating the rotation and direction attributes of objects into the IoU metric. The RWIoU loss can replace the Lnorm and direction losses to help detectors achieve higher accuracy. Finally, a 3D object detection framework dubbed DCDet is proposed which combines the DCLA and RWIoU.

The contributions of this work are summarized as follows:

• We thoroughly investigate the current widely used label assignment strategies and analyze their pros and cons. Based on experimental observations, we introduce a new label assignment strategy called dynamic cross label assignment (DCLA).
• We propose a rotation-weighted IoU (RWIoU) to better measure the proximity of two rotation boxes compared to the L1 metric. RWIoU takes rotations and directions of 3D objects into consideration simultaneously.
• A 3D object detector dubbed DCDet is proposed which combines the DCLA and RWIoU. Extensive experiments on the Waymo Open [Sun et al., 2020] and KITTI [Geiger et al., 2012] datasets demonstrate the effectiveness and generality of our methods.

2 RelatedWork

2.1 3DObject Detection

VoxelNet [Zhou and Tuzel, 2018] encodes voxel features using PointNet [Qi et al., 2017a], and then extracts features from 3D feature maps through 3D convolutions. SECOND [Yan et al., 2018] efficiently encodes sparse voxel features by proposed 3D sparse convolution. PointPillars [Lang et al., 2019] divides a point cloud into pillar voxels, avoiding the use of 3D convolution and achieving high inference speed. 3DSSD [Yang et al., 2020] significantly improves inference speed by discarding upsampling layers and refinement networks commonly used in point-based methods. PointRCNN [Shi et al., 2019] produces proposals from raw points using PointNet++ [Qi et al., 2017b], and then refines bounding boxes in the second stage. PV-RCNN [Shi et al., 2020a] uses features of internal points to refine proposals. Voxel RCNN [Deng et al., 2021] replaces the features of raw points in the second-stage refinement with 3D voxel features in the 3D backbone.

2.2 Label Assignment

Label assignment, which is fundamental to 2D and 3D object detection, significantly influences the optimization of a network. Its development is more mature in 2D object detection, with RetinaNet [Lin et al., 2017] assigning anchors on the output grid map, FCOS [Tian et al., 2019] designating grid points within the range of ground truth boxes as positive samples, and CenterNet [Zhou et al., 2019b] identifying center points of ground truth boxes as positive samples. ATSS [Zhang et al., 2020] and AutoAssign [Zhu et al., 2020] propose adaptive strategies for dynamic threshold selection and dynamic positive/negative confidence adjustment, respectively. YOLOX [Ge et al., 2021] introduces the SimOTA scheme for dynamic positive sample selection. Conversely, 3D object detection label assignment is less developed, grappling with unique challenges such as maintaining a balance of positive samples across various object sizes. Current methods in 3D object detection typically use either anchor-based [Yan et al., 2018; Lang et al., 2019; Deng et al., 2021] or center-based [Yin et al., 2021; Ge et al., 2020; Hu et al., 2022] label assignment schemes. However, these schemes have drawbacks: the anchor-based label assignment often results in unbalanced assignments, and the center-based label assignment may overlook high-quality samples. To simultaneously overcome the above two drawbacks, we propose the dynamic cross label assignment (DCLA). Details about the DCLA are described in the methodology section.

2.3 IoU-based Loss

IoU-based losses [Rezatofighi et al., 2019; Zheng et al., 2020; Zhang et al., 2022a] without rotation have been well studied in 2D object detection. These methods not only ensure consistency between the training objective and the evaluation metric but also normalize object attributes, leading to enhanced performance compared to the Lnorm loss. Due to their success in 2D object detection, some 3D object detection methods [Zhou et al., 2019a; Sheng et al., 2022; Shi et al., 2022] incorporate IoU-based losses. 3DIoU [Zhou et al., 2019a] extends IoU calculation from 2D to 3D by considering rotation. However, the optimization direction of 3DIoU-based loss can be opposite to the correct direction. To address this, RDIoU [Sheng et al., 2022] decouples rotation from 3DIoU. It considers rotation as an attribute similar to object location, but it doesn't consider object direction. A direction loss is needed for classifying object directions. ODIoU [Shi et al., 2022] combines L1 metric and axis-aligned IoU to regress objects. Our proposed RWIoU incorporates both rotation and direction into the IoU metric, eliminating the need for Lnorm and direction losses. Details of RWIoU will be explained in the next section.

Inference

Figure 2: The overall framework of our DCDet. The dynamic cross label assignment scheme is only used in the training phase.

3 Methodology

This section will describe the dynamic cross label assignment (DCLA) and the rotation-weighted IoU (RWIoU) in detail. The overall framework is illustrated in Figure 2.

3.1 Dynamic Cross Label Assignment

The label assignment schemes used in existing 3D object detection methods are generally based on prior information such as spatial ranges or object scales to manually select positive samples. For example, the anchor-based label assignment uses object scales to set the sizes of anchors and then uses anchors with IoU greater than a certain threshold as positive samples. The anchor-based label assignment generally produces unbalanced positive samples for different-scale objects, causing the model to prioritize large-scale objects. The center-based label assignment usually takes the center points of grounding truths as positive samples. This can result in a large number of good-quality samples being discarded, leaving inefficient utilization of training data.

The above label assignment schemes have a common property, they all use static prior information as the selection criteria. And the prior information is determined by human experience. Dynamic label assignment schemes [Zhang et al., 2020; Zhu et al., 2020; Ge et al., 2021] have shown their advantages in 2D object detection. However, directly transferring these schemes to 3D object detection is not trivial. There are some challenges: 1) There is no space to dynamically select positive samples for small objects (e.g. pedestrians). Because small objects generally cover one or two grid points on the output map; 2) The coverage of objects with different scales varies greatly. This easily results in an imbalance of positive samples between different scale objects.

To dynamically select sufficient high-quality positive samples while maintaining the balance between different scale objects, we propose a dynamic cross label assignment (DCLA) scheme. Specifically, it limits the positive sampling range in a cross-shape region for each object. Typically, an object's center region on a feature map contains enough features to identify it [Tian et al., 2019], and objects in point clouds have regular shapes. Therefore, we only use the center point and its surrounding points for positive sampling in the DCLA scheme. We refer to this sampling range as the cross region. It can be adjusted by a parameter r to adapt to outputs with different grid cell sizes as illustrated in Figure 1. r is the Manhattan distance away from the center point. When r = 1, the cross region covers the center and its top, down, left, and right neighbors. And when r = 0, the DCLA degenerates to the center-based label assignment.

The implementation steps of the DCLA are described in detail next. Given a ground truth bt and positions P in its cross region, calculate the selection cost as follows:

where the Lcls j and Lreg j are the classification loss and regression loss between the ground truth bt and j-th prediction boj respectively, and λreg is the weight of regression loss. Then, sort the predictions in the cross region according to the selection costs. Next, sum the IoUs between the ground truth bti and predictions boj , j ∈ P :

We utilize k as the number of positive samples for ground truth bt . Finally, select the top k predictions as positive samples. And the rest predictions are negative samples.

Specifically, given a point cloud input and the ground truth boxes {bt1 , bt2 , · · · , btn }, we assume that f(bti , boij ) represents the regression loss function, where bti and boij denote the i-th ground truth box and its j-th predicted box, respectively. Therefore, the regression loss ℓ for the point cloud is calculated as follows:

where N represents the total number of positive samples in the input point cloud, and ki denotes the number of positive samples assigned for the ground truth bti . Notably, ki is calculated independently for each ground truth, as in Eq. (2). It is related to the number of high-quality samples in the cross region and is not dependent on the ground truth scale. However, in the anchor-based label assignment, ki varies significantly with the ground truth scale, resulting in a bias towards large-scale objects in the loss. For the center-based label assignment, ki is always equal to 1, leading to inefficient utilization of training data.

We adopt the heatmap target for the classification task. The weights of positive samples are set to 1, and the weights of negative samples in cross regions are set to the values of IoU between predicted boxes and ground-truth boxes. As for the rest negative samples, the weights are all set to 0.

3.2 Rotation-Weighted IoU

In general, different object categories exhibit significant scale variations, and various attributes such as location, size, and rotation also possess scale differences. Many existing methods employ the Lnorm loss as the regression loss. However, this loss function renders the model sensitive to differences in both object and attribute scales. Consequently, large objects and attributes dominate the total loss. The IoU metric can normalize object attributes, making it immune to scale differences. Moreover, the optimization objective of the IoU-based loss aligns with the evaluation metrics of detection models. Therefore, substituting the Lnorm loss with the IoU-based loss often yields accuracy improvement [Zheng et al., 2020; Rezatofighi et al., 2019; Sheng et al., 2022].

Utilizing IoU-based loss in 3D object detection poses several challenges. Firstly, calculating traditional IoU requires the computation of polyhedron volumes, which is a complex and computationally expensive task. Secondly, the traditional IoU-based loss, due to its tight coupling with rotation, can lead to misdirection in optimization, resulting in training instability [Sheng et al., 2022]. Lastly, integrating the traditional IoU metric with object directions is not trivial. Therefore, the inclusion of L1 loss or direction loss becomes necessary to aid models in classifying object directions.

To tackle the aforementioned challenges, we propose a rotation-weighted IoU (RWIoU). It thoroughly decouples the rotation from the IoU calculation, making the computation similar to the axis-aligned IoU computation. RWIoU can be implemented with just a few lines of code. By integrating sine and cosine values of rotations of objects into a rotation weighting item, our RWIoU can penalize rotation and direction errors simultaneously.

The RWIoU calculation process is shown in Figure 3. It first considers two rotation boxes B1 and B2 as axis-aligned boxes, and then calculates the intersecting volume of the two axis-aligned boxes as follows:

where (xi, yi, zi), i ∈ {1, 2} denote the locations of box centers, (li,wi, hi), i ∈ {1, 2} represent the sizes of boxes, and Vinter denotes the intersecting volume of two axis-aligned boxes. Then, we update the Vinter according to the rotation difference of the two boxes as follows:

where θ1 and θ2 represent rotations of two boxes, ωs and ωc denote the sine and cosine rotation error factor respectively that are all normalized to the range of [0, 1], ω represents the rotation weighting item, Vweighted is the rotation-weighted value of Vinter, α ∈ [0, 1] is a hyper-parameter which is used to control the contribution of rotation to the RWIoU. If α = 0, the RWIoU degrades to axis-aligned IoU. After obtaining Vweighted, the value of RWIoU can be calculated as follows:

where V1 and V2 represent the volumes of two boxes, respectively. The gradient analysis of the RWIoU is in Appendix.

3.3 Loss Function

Single-stage detectors typically encounter misalignment between classification confidence and localization accuracy. To solve the misalignment problem, we follow Zheng et al. to introduce an extra IoU prediction branch. The classification loss Lcls and IoU prediction loss Liou are the same as those of CIA-SSD [Zheng et al., 2021].

The regression loss Lreg is based on the RWIoU. It is calculated as follows:

where N is the total number of positive samples, RWIoUi and Di represent the RWIoU value and the L2 distance of centers, respectively. Additionally, Diagi denotes the diagonal length of the minimal enclosing rectangle of the i-th predicted box and its ground truth. The term Di Diagi is used to optimize the prediction of center locations. Since our RWIoU incorporates sine and cosine functions to represent the rotation angle of a bounding box, the need for a direction loss is eliminated. The overall loss function is calculated as follows:

where λcls, λreg, and λdir are the weight of classification, regression, and direction losses, respectively.

4 Experiments

In this section, we evaluate models on widely-used 3D object detection benchmark datasets including Waymo Open [Sun et al., 2020] and KITTI [Geiger et al., 2012].

4.1 Implementation Setup

Data Preprocessing

For the Waymo Open dataset, the detection range is [-74.88, 74.88]m for the X and Y axes and [-2, 4]m for the Z axis, the voxel size is set to (0.08, 0.08, 0.15)m. For the KITTI dataset, the detection range is [0, 70.4]m for the X axis, [-40, 40]m for the Y axis, and [-5, 3]m for the Z axis, the voxel size is set to (0.05, 0.05, 0.1)m.

Training Details

The backbone of our DCDet is the same as that of CenterPoint [Yin et al., 2021]. Following PillarNeXt [Li et al., 2023], we use a feature upsampling in the detection head of DCDet, which increase the output resolution with only a little overhead. All models are trained from scratch in an end-to-end manner with the Adam optimizer and a 0.003 learning rate. And the parameter α used in Eq. (4) is set to 0.5. The parameters λcls and λiou used in Eq. (7) are all set to 1. And the parameter λreg used in Eq. (1) and Eq. (7) is set to 3. For the Waymo Open and KITTI datasets, the parameter r used in DCLA is set to 1 and 3, respectively. On the Waymo Open and KITTI datasets, models are trained for 30 epochs with a batch size of 24 and 80 epochs with a batch size of 8, respectively. Hyper-parameters analysis is in Appendix.

4.2 Comparison with State-of-the-Art Methods

The baseline models presented in Table 1 primarily utilize either center-based or anchor-based label assignment. Moreover, they commonly employ Lnorm regression loss. As depicted in Table 1, the center-based label assignment demonstrates a significant advantage over the anchor-based label assignment on the Waymo Open dataset. Nevertheless, our DCDet, featuring a lightweight single-stage network, surpasses the state-of-the-art center-based method DSVT, which employs a heavy backbone network. Notably, even our DCDet model trained on only 20% of the training samples outperforms both the center-based and anchor-based methods trained on the entire dataset. These results demonstrate the superior performance of our DCDet framework which employs DCLA and RWIoU-based regression loss.

We also evaluated our DCDet on the Waymo Open test set by submitting the results to the official server. The performance comparisons are presented in Table 2, revealing that our DCDet surpasses previous state-of-the-art methods significantly. Particularly, in the case of small-scale categories such as pedestrians and cyclists, our method demonstrates a substantial advantage due to the balanced and sufficient positive samples provided by DCLA.

4.3 Effect on Different Backbone Networks

To assess the generality of our DCLA and RWIoU, we conduct experiments by incorporating them into several widely used backbone networks, namely SECOND, PillarNet, and DSVT. All models are reproduced using the OpenPCDet [Team, 2020] codebase. We train these models using both 20% and 100% of the training data from the Waymo Open dataset and present the results in Table 3. As evident from the table, the integration of our DCLA and RWIoU yields significant improvements across all model groups. This underscores the generality and effectiveness of our proposed DCLA and RWIoU techniques. Notably, the DCLA and RWIoU-based regression loss belong to the learning strategies of models, resulting in cost-free improvements. Even when trained on only 20% of the training data, the models integrated with our DCLA and RWIoU techniques either surpass or catch up to the performance of models trained on the entire training data without these enhancements. This demonstrates that our learning strategies enhance the utilization of training data, which is particularly valuable considering the high cost associated with labeling 3D bounding boxes.

4.4 Ablation Study

To further study the influence of each component of DCDet, we perform a comprehensive ablation analysis on the Waymo Open and KITTI datasets. For the Waymo Open dataset, following prior works [Shi et al., 2020a; Wang et al., 2023], models are trained on 20% training samples and evaluated on the whole validation samples. For the KITTI dataset, models are trained on the train set and evaluated on the val set.

Effect of RWIoU and DCLA

The baseline model adopts center-based label assignment and L1 regression loss. To evaluate the effectiveness of our proposed methods, we systematically integrate RWIoU-based regression loss and DCLA into the baseline model. The ablation results are presented in Table 4. We observe a notable performance improvement when incorporating RWIoU-based regression loss, as demonstrated by the results in the 1st and 2nd rows of Table 4. This suggests that the proposed loss function is better suited for the task of 3D object detection compared to the traditional L1 loss. Furthermore, models trained with DCLA consistently achieve significantly better performance than the baseline, as illustrated in the 1st and 3rd rows of Table 4. This indicates that DCLA facilitates improved utilization of the available training data, thus enhancing the overall model performance. Notably, when both RWIoU-based regression loss and DCLA are used, the model achieves the highest performance among all evaluated models. These findings validate the effectiveness of our proposed methods and highlight the importance of carefully designing the loss function and label assignment for improving the performance of 3D object detectors.

Comparison with Other Regression Losses

Table 5 provides a comparison of different regression losses. All models utilize the DCLA scheme and the same backbone network. The results in the 1st , 2nd , and 3rd rows of Table 5 reveal marginal differences between the L1, RDIoUbased [Sheng et al., 2022], and ODIoU-based [Shi et al., 2022] regression losses. However, our RWIoU-based loss exhibits a significant performance improvement compared to the other regression losses, as demonstrated in the 4th row of Table 5. These results highlight the effectiveness of our RWIoU, which decouples the rotation from IoU calculation by introducing rotation weighting. Notably, the RDIoUbased loss necessitates an additional direction classification loss, and the ODIoU-based loss requires an extra L1 loss. In contrast, our RWIoU-based loss is a pure IoU-based loss without any auxiliary losses. This simplification allows our approach to achieve superior performance without introducing additional complexity.

Comparison with Other Label Assignment Schemes

Table 6 compares different label assignment schemes with all models using the RWIoU-based regression loss and the same backbone network. As depicted in the 1st and 3rd rows of Table 6, both anchor-based and box-based label assignment exhibit subpar performance when it comes to small objects like pedestrians and cyclists. This is mainly due to the unbalanced assignment of positive samples for objects with different scales. On the other hand, the center-based label assignment, as shown in the 2nd row of Table 6, achieves good results on the Waymo Open dataset but performs poorly on the KITTI dataset. We argue that this discrepancy arises from overlooking a large number of excellent samples, resulting in an insufficient number of positive samples for training on small-scale datasets like KITTI. Moreover, the poor performance of simOTA [Ge et al., 2021] in 3D object detection, as demonstrated in the 4th row of Table 6, highlights the challenges of directly transferring methods from the 2D domain to the 3D domain. However, our DCLA outperforms these baseline label assignment schemes on both the Waymo Open and KITTI datasets, as illustrated in the last row of Table 6. This confirms that our DCLA can adapt to datasets of different scales by enabling balanced and adequate positive sampling.

5 Conclusion

In this paper, we propose a dynamic cross label assignment (DCLA), which dynamically assigns positive samples from a cross-shaped region for each object. The DCLA scheme mitigates the imbalanced issue in the anchor-based assignment and the loss of high-quality samples in the center-based assignment. Thanks to the balanced and adequate positive sampling, DCLA effectively adapts to different scale datasets. Moreover, a rotation-weighted IoU (RWIoU), which considers the rotation and direction in a weighting way, is introduced to measure the proximity of two rotation boxes. Extensive experiments conducted on various datasets demonstrate the generality and effectiveness of our methods.

Acknowledgments

This work is supported by the Project of Guangxi Key R & D Program (No. GuikeAB24010324).