Tracklet Proposal Network for Multi-Object Tracking on Point Clouds

1 Introduction

Multi-Object Tracking (MOT), which is a key technology for extracting dynamic information from the environment, has wide applications in autonomous driving and robotics. Most existing methods explore MOT on images [Hu et al., 2019; Mykheievskyi et al., 2020] or LiDAR point clouds [Simon et al., 2019; Weng et al., 2020a]. These methods usually follow a tracking-by-detection paradigm that associates the detected objects across frames, and their tracking performance is often restricted by inevitable detection failures (e.g., due to heavily occluded objects).

To tackle detection failures, in image-based MOT, previous methods [Choi, 2015; Zhang et al., 2020] use short trajectories (tracklets), composed by multiple detections in a short time, as a basis for object association to form longer trajectories. Recently, [Sun et al., 2020] directly generates tracklets from image sequences in an end-to-end manner, and achieves state-of-the-art performance in image-based MOT. For point cloud-based MOT, however, such an end-to-end tracklets-based network design is underexplored. The main challenge here is how to produce accurate tracklets from irregular and unordered point clouds, as more accurate tracklets will lead to higher MOT performance. [Luo et al., 2018] proposes an end-to-end network that decodes tracklets by averaging current detections and previous predictions; however, its accuracy is sensitive to the quality of the predictions.

∗Corresponding author

The accuracy of a tracklet relies on the quality of detections inside it. In a tracklet, an object across consecutive frames shares some consistent spatial-temporal features. For example, in point clouds, the width, length, and height of rigid objects (e.g., vehicles) are consistent in consecutive frames. Even for non-rigid objects (e.g., pedestrians), the size, volume, and geometry are often similar in consecutive point cloud frames. Such consistency constraints in spatialtemporal features can improve the quality of detections. We investigate to learn and use such features to improve tracklet accuracy by refining the quality of detections inside tracklets.

The proposal-based methods have been successfully applied in object detection [Shi et al., 2020], single object tracking [Li et al., 2019] and action detection [Hou et al., 2017] by performing object localization and recognition in a coarse-tofine manner. Through such a proposal-based coarse-to-fine refinement, we could obtain more accurate tracklets, and consequently, better tracking objects in consecutive frames.

Inspired by this, we develop a novel Tracklet proposal Convolutional Neural Network, named PC-TCNN, for MOT on Point Clouds. Our PC-TCNN performs MOT by (1) generating tracklet proposals, (2) refining tracklet proposals, and then (3) associating the refined tracklets, as shown in Figure 1. Specifically, it first produces a set of high-recall tracklet proposals as regions of interest by performing object proposal generation and motion regression on point cloud sequences. Then, PC-TCNN employs a tracklet proposal refinement scheme to refine the tracklet detections (detections inside a tracklet) using spatial-temporal features aggregated from each tracklet proposal, thereby improving tracklet accuracy. Furthermore, to form long trajectories, our PC-TCNN associates the refined tracklets using a greedy assignment. Our contributions are summarized as follows:

• We propose the first tracklet proposal network, which generates tracklet proposals, refines the proposals, and associates the refined tracklets on point cloud sequences, leading to significantly improved MOT performance;
• We introduce a tracklet proposal refinement scheme that uses the consistency constraints in spatial-temporal features of each tracklet proposal to refine the tracklet detections, and this can greatly enhance the accuracy of tracklets;
• Our method achieves the state-of-the-art on the KITTI tracking benchmark [Geiger et al., 2012] and ranks 1st among all submitted methods as of Jan. 20th, 2021.

2 RelatedWork

Tracking-by-detection on point clouds. Most existing MOT methods on point clouds follow a tracking-by-detection paradigm. They first detect objects from single frame, then associate the objects across frames by applying a filtering algorithm [Simon et al., 2019; Weng et al., 2020a]. Some work constructs deep networks to associate detected objects [Baser et al., 2019; Wang et al., 2020]. Recent studies also investigate MOT using both point clouds and image data [Weng et al., 2020b]. In these methods, the detection is separated from tracking. Hence, their tracking performance is restricted by inevitable detection failures.

Tracklet-based MOT. To tackle detection failures, some previous image-based MOT methods [Bae and Yoon, 2014; Choi, 2015; Zhang et al., 2020] first generate tracklets by independently detecting and matching similar objects in consecutive image frames, then associate the tracklets to generate long trajectories. These methods are less sensitive to detection failure. Recently, [Sun et al., 2020] proposed a DMM-Net that generates tracklets from images in an end-toend manner. [Luo et al., 2018] proposed an end-to-end FaF network that merges current detections and past predictions to decode tracklets on point clouds; however, its accuracy relies on the prediction quality. Our method directly exploits feature consistency of objects in consecutive frames to improve the detection accuracy, leading to more accurate tracklets.

Proposal-based methods. The proposal-based methods, which usually contain proposal generation and proposal refinement stages, perform through a coarse-to-fine framework.

The region proposal-based methods, specifically, generate regions of interest in the spatial domain, and have been successfully applied for 2D object detection [Girshick et al., 2014], single 2D object tracking [Li et al., 2019] and 3D object detection [Shi et al., 2019; Shi et al., 2020]. These methods were designed for single frame data and do not use temporal features. Recently, tube proposal-based methods, which models features by taking into account temporal information, have been used in action detection [Hou et al., 2017] and recognition [Wu et al., 2020]. These models were developed on regular image data rather than irregular point clouds. To our best knowledge, our method is the first tracklet proposal framework on point cloud sequences for MOT.

3 Our Method

We illustrate our tracklet proposal network in Figure 2. The framework consists of three modules: tracklet proposal generation, tracklet proposal refinement, and tracklet association.

3.1 Tracklet Proposal Generation

To localize spatial-temporal region of interest in point clouds, which we denote as tracklet candidates or tracklet proposals, we propose a tracklet proposal generation module. At each timestamp t, the proposed module takes multiple point cloud frames {Pi}ti=t-n (n past frames and one current frame) as inputs, and encodes the input point clouds into multiple bird view feature maps. Then, it performs 3D object proposal generation and motion regression on these feature maps to generate a set of high recall tracklet proposals.

Backbone Network

To efficiently convert point cloud sequences into 3D feature volumes, we employ the 3D sparse convolution [Shi et al., 2020] to obtain the spatial features. Specifically, we first register the input frames to the coordinate system of the last frame using the GPS/IMU data. The registered point cloud frames are denoted by {Pi}0i=-n . Then, we voxelize multiple point cloud frames with a spatial resolution of L×W ×H (length, width and height), and obtain voxels {P ′ i }0i=-n . For each voxel, we calculate the raw features using mean of a 4tuple (3D coordinates + intensities) raw features of all inside points. Then, we apply a series of shared 3D sparse convolution S(·) on the voxelized point cloud frames to obtain the spatial features {F 3D i }0i=-n by

Here S(·) consists of a series of 3×3×3 3D sparse convolution kernels, which downsample the spatial features to 1×, 2×, 4×, and eventually a 8× downsampled tensor.

Inspired by the convolutional gated recurrent unit (ConvGRU) [Tokmakov et al., 2017], which has been proven to be effective in modeling temporal features in 2D images, we incorporated it into our tracklet proposal generation module to compute temporal features of point clouds. We first compress the spatial features {F 3D i }0i=-n along the H dimension into multiple bird's eye view (BEV) feature maps {F BEV i }0i=-n . Then we apply the ConvGRU G(·) on BEV feature maps to compute the temporal features {F T i }0i=-n , by

where F T-n-1 are initialized to zeros.

The temporal features are further encoded into high-level BEV features {F H i }0i=-n (which represent more comprehensive spatial-temporal features of point clouds) by a series of shared 2D convolution C(·),

Tracklet Proposal Head

A tracklet in point clouds is defined by multiple 3D bounding boxes (BBs), which encode objects' 3D sizes, motions and correspondences during a short period. We denote the 3D BB of j-th tracklet proposal in i-th frame as Bj i = [xji , yj i , zj i ,wj i , hji , lj i , αj i ], which includes object coordinates, width, height, length and orientation angle. Then the j-th tracklet proposal is given by T j = {Bj i }0i=-n .

The tracklet proposal head takes the BEV feature map sequence {F H i }0i=-n as inputs. At each location (pixel) on each feature map, we employ an object proposal head to compute the object confidence and residuals, and construct a motion head to compute a cross-frame-offset of object motion. We employ non-maximum suppression (NMS) on the last frame of BB proposal head, to obtain the seeds of tracklet proposal. Starting from the tracklet proposal seeds, we generate the m tracklet proposals, denoted by Tt = {T j |j = 0, . . . ,m}, based on the cross-frame-offsets.

3.2 Tracklet Proposal Refinement

To improve the quality of tracklet detections, which is essential in attaining more accurate tracklets, we propose a tracklet proposal refinement module. In most proposal-based approaches [Shi et al., 2020; Hou et al., 2017], a main challenge in the refinement stage is how to effectively extracting features from area of interest. Previous feature extraction methods (e.g., region of interest pooling and tube of interest pooling) are mostly designed for single frame [Girshick et al., 2014; Shi et al., 2020] or video data [Hou et al., 2017] and cannot be directly applied on point cloud sequences. To tackle this, we design a tracklet feature aggregation method to capture features from spatial-temporal region of interest on point cloud sequence, leading to more accurate object recognition and localization in tracklets.

Tracklet Features Aggregation

We first extract the points of each tracklet proposal by employing context-aware point cloud pooling [Shi et al., 2019]. For i-th 3D BB Bj i in the j-th tracklet proposal, we randomly extract b object points from input point cloud frame Pi. These points are denoted by P j i = {pj,k i |k = 0, . . . , b}, where p ∈ R3 indicates point coordinates.

The extracted tracklet points contain accurate local object features inside BB, but lack surrounding and global features that are outside. Therefore, we project each point pj,k i to the coordinate system of backbone features F T i , and utilize bilinear interpolation to obtain new backbone features f j,k i . To exploit more accurate local geometry, we transform the extracted points to the canonical coordinate system of the corresponding BB, denoted by ˆ pj,k i . Thus, for i-th 3D BB in the j-th tracklet proposal, we obtain a set of extracted features F j i = {[ˆ pj,k i , f j,k i ]|k = 0, . . . , b}. These features are aggregated by set abstraction [Qi et al., 2017] layers A(·) with single-scale grouping to generate a feature vector ˆ F j i = A(F j i ).

In a tracklet, an object across consecutive frames usually shares some consistent spatial-temporal features. Especially for the rigid objects, their sizes are the same across frames.

To enhance their relation, we further aggregate the features ˜ F j i inside the proposal by employing a gated recurrent unit (GRU) layer R(·) as

where ˜ F j-n-1 are initialized by zeros.

Finally, we obtain a sequence of aggregated features for each j-th tracklet proposal, denoted by { ˜ F j i }0i=-n , which are used to perform tracklet refinement.

Refinement Head

We refine a tracklet proposal T j by refining its multiple 3D BBs {Bj i }0i=-n . During training, we assign the ground-truth tracklet to the proposal T j if the 3D overlap between the BBs in ground-truth tracklet and in proposal is larger than a threshold. For each Bj i , we conduct box refinement by directly regressing the size, location, and orientation residuals relative to it using the aggregated features ˜ F j i . For each tracklet proposal, we also compute a confidence using the features ˜ F j 0 ∈ { ˜ F j i }0i=-n . The training target of the confidence is formulated as min(1, max(0, 2mIoU - 0.5)), where the mIoU is calculated by the mean 3D intersection between {Bj i }0i=-n and their ground-truth. The confidence loss is formulated to minimize the cross-entropy loss between confidence outputs and training targets.

In the inference stage, we perform NMS on the 3D BBs in the last frame to obtain a set of refined tracklets ˆ Tt, which encodes more accurate object sizes, motions, and correspondences during a short period of time.

3.3 Tracklet Association

Inspired by 2D image-based tracklet association which generates long trajectory by associating the tracklets with previous trajectory set using their 2D IOUs [Sun et al., 2020], we transform this strategy into 3D trackelt association. We first initialize an empty trajectory set at the beginning of tracking. Then, at each timestamp t, we associate our refined tracklets ˆ Tt with previous trajectories using a simple greedy matching algorithm based on their 3D IoUs. For successfully associated tracklets, we use them to update the corresponding trajectories. Unsuccessfully associated tracklets will initialize new trajectories for MOT at the next timestamp. This proposed method is able to track objects under both online and global settings. The tracklets encoded by multiple detections can recover missed objects across multiple frames.

3.4 Losses

The proposed PC-TCNN is trained in an end-to-end manner with a tracklet proposal loss Ltpn and a proposal refinement loss Ltrn . Similar to [Shi et al., 2020], we sum the two losses with equal weights to generate a final training loss as L = Ltpn + Ltrn . Ltpn consists of multiple object losses Lobj i and motion losses Lmot i weighted by β,

We employ the object loss Lobj i in i-th frame following single frame object detector [Shi et al., 2020],

where r ∈ {xi, yi, zi, wi, hi, li, αi}; the smooth L1 loss Lsl1 is used for single frame anchor box regression with the network output residual ∆̂ rp and regression target ∆rp ; focal loss Lcls is employed for anchor classification. We also adopt the smooth L1 loss on the predicted motion residuals ∆̂ rm and motion targets ∆rm to regress the object motion as

The Ltrn loss consists of multiple object refinement residual losses and a confidence loss as

where r ∈ {xi, yi, zi,wi, hi, li, αi}; the confidence loss LmIoU is calculated by a cross-entropy loss with confidence outputs and confidence targets as detailed in Section 3.2; The residual loss is formulated by smooth L1 loss with the network output residuals ∆̂ rs and the target residuals ∆̂ rs .

4 Experiments

4.1 Datasets and EvaluationMetrics

We evaluate our method on the KITTI tracking dataset [Geiger et al., 2012], which consists of 21 training sequences and 29 test sequences. Input data of experiments include point clouds and IMU/GPS data. The images are only used to visualize tracking results. Same as most prior works, we report results on the car subset for comparison.

Since the evaluation of the KITTI benchmark is conducted on 2D image plane, we projected our 3D tracking results on image plane for evaluation. We adopted the widely used CLEAR MOT metrics [Bernardin and Stiefelhagen, 2008] (including MOTA, MOTP, IDS, and FRAG) and Mostly Tracked (MT) / Mostly Lost (ML) [Li et al., 2009] metrics for MOT evaluation. We also report the results evaluated by the 3D MOT metrics proposed by [Weng et al., 2020a].

4.2 Implementation Details

In our experiments, the 3D sparse convolution has four layers with feature dimensions (16, 32, 64, 64). Feature dimension of the ConvGRU and 2D convolution is 128 and 256, respectively. We set the tracking range within [0, 70.4]m for the X axis, [-40, 40]m for the Y axis, and [-3, 1]m for the Z axis. The input point clouds are voxelized along X, Y, Z with a size (0.05m, 0.05m, 0.1m), respectively. We employed the set abstraction layers with grouping scales 128, 32 and 1. During training, we performed NMS with a 3D IoU threshold of 0.8 to randomly keep 64 tracklet proposals with 1:1 negative and positive proposals. A tracklet proposal that has mean 3D IoU with ground-truth tracklet beyond 0.5 is treated as a positive proposal. Otherwise, it is treated as a negative proposal. During inference, we performed NMS with a threshold of 0.7 to keep top-100 proposals. Also, we used an NMS threshold of 0.1 to remove the redundant tracklets.

To avoid overfitting, we adopted a series of data augmentation on the registered point cloud frames. It includes random flipping along the X axis, global scaling with a random factor in [0.95,1.05], global rotation around the Z axis with a random angle in [- π2 , π2 ]. We aslo performed data augmentation on tracklets by randomly sampling ground-truth tracklets from other training samples to current training samples to simulate objects' locations and motions in various environments. We adopted the ADAM optimizer with batch size 4, learning rate 0.01 and 80 epochs on two RTX 2080 Ti GPUs to train our PC-TCNN. We employed the onecycle learning rate strategy to deal with the learning rate decay.

4.3 Ablation Study

We conducted a number of experiments to examine each component/design of the proposed PC-TCNN. For the ablation study, we split the KITTI training set into two sub-datasets for training and validation. The sub training set consists of 11 sequences, and the validation set consists of 10 sequences.

Number of input frames. We conducted an ablation study on the number of input frames. The results are shown in Table 1. On the validation set, our method achieved the best MOT performance when taking four frames as inputs. Although the false negatives can be further reduced by taking more frames, the false positives dramatically increase with more background noise being introduced. The overall inference time also becomes longer when the number of input frame increases as more computation is required. Some strategy can speed up the network, such as enlarging the voxel size, but may lower tracking performance.

Backbone network. To investigate the settings of the backbone network, we compared our settings of 3D sparse convolution (SpConv)+ConvGRU with a baseline setting of 2D Conv+3D Conv used in FaF [Luo et al., 2018]. The results are shown in Table 2. When both pipelines take in four frames, our method outperforms the baseline by 1.83% on MOTA, suggesting that the 3D SpConv+ConvGRU backbone is an essential setting in our method.

Tracklet augmentation. To analyze the effectiveness of the tracklet augmentation, we removed the tracklet augmentation and kept other components unchanged. The results are shown in Table 2. We can see that without tracklet augmentation, the MOT performance drops significantly (2.14%), indicating the benefit of tracklet augmentation.

Tracklet proposal refinement scheme. We further investigated the effectiveness of the tracklet proposal refinement scheme. As shown in Table 2, with tracklet refinement, the average precision of tracklet detection in the last frame gains a 3.47%, demonstrating that our tracklet proposal refinement improves the tracklet accuracy. As shown in the same table, the tracklet proposal refinement scheme gains a 3.62% on MOTA. This demonstrates the proposed tracklet refinement can further improves MOT on point cloud sequences.

4.4 Results on KITTI Tracking Benchmark

The car tracking results on the leaderboard of the KITTI tracking benchmark are summarized in Table 3 (only recently published methods are reported). The PC-TCNN takes four point cloud frames as inputs, and is trained on 20 sequences of the KITTI training set and validated on the remained one sequence. Our method, achieves an MOTA of 91.75%, currently ranks 1st among all submitted methods as of Jan. 20th, 2021. Our method also achieves a high recall of 96.08%, outperforms the strongest prior model by 2.88%. The outstanding performance is due to that the tracklet proposal design greatly enhances the tracklet accuracy by better modeling spatial-temporal features of point cloud sequences. These tracklets, encoded by accurate detections and motions, significantly reduce the false negatives, ID switches, and fragments in the point cloud-based MOT.

We also show our qualitative results on two sequences of the KITTI test set in Figure 3. For better visualization, we rendered the 3D BBs using CAD car models. We also showed a long-time tracking example in Figure 4, in which the object with ID 2 is tracked over 340 meters, although it is heavily occluded by the object ID 101 in many frames.

4.5 Evaluating in 3D Space

For a broader comparison with 3D MOT methods, we also conducted experiments using the data split as GNN3DMOT [Weng et al., 2020b]. The results on the validation set are shown in Table 4, where the sAMOTA, AMOTA, IDS and FRAG are 3D MOT metrics proposed by AB3DMOT [Weng et al., 2020a]. Our method outperforms the GNN3DMOT 1.76% in the most important sAMOTA metric, and also surpasses the FANTrack [Baser et al., 2019], AB3DMOT [Weng et al., 2020a] and GNN3DMOT [Weng et al., 2020b] in the AMOTA and FRAG metrics, further demonstrating the effectiveness of our method.

5 Conclusion

We present a novel tracklet proposal PC-TCNN network for MOT on point clouds. By innovatively employing a proposalbased framework, our PC-TCNN performs tracklet proposals generation, refinement, and association, leading to significantly improved MOT performance. Meanwhile, we introduced a novel tracklet proposal refinement scheme that extracts and aggregates spatial-temporal features of each proposal to refine the tracklet detections, and this greatly enhances the accuracy of tracklets. With these novel designs, our network achieved better MOT performance over the stateof-the-art methods.