Structure Guided Lane Detection

1 Introduction

Lane detection, which aims to detect lanes in road scenes, is a fundamental perception task and has a wide range of applications (e.g., ADAS [Butakov and Ioannou, 2014], autonomous driving [Chen and Huang, 2017] and high-definition map production [Homayounfar et al., 2019]). Over the past years, lane detection has made significant progress and it is also used as an important element for tasks of road scene understanding, such as driving area detection [Yu et al., 2020].

To address the task of lane detection, lots of learningbased methods [Pan et al., 2018; Qin et al., 2020] have been proposed in recent years, achieving impressive performance on existing benchmarks [TuSimple, 2017; Pan et al., 2018]. However, there still exist several challenges that hinder the development of lane detection. Frist, there lacks a unified and effective lane representation. As shown in (a) of Fig. 1, there exist various definitions including point [TuSimple, 2017], mask [Pan et al., 2018], marker [Yu et al., 2020] and grid [Lee et al., 2017], which are quite different in form for different scenarios. Second, it is difficult to model the structural relationship between scenes and lanes. As displayed in (b) of Fig. 1, the structural information depending on scenes, such as location of vanishing points and parallelism of lanes, is very useful, but there is no scheme to describe it. Last, while predicting lanes, it is also important to predict other attributes including instance and type (see (c) of Fig. 1), but it is not easy to extend these for existing methods. These three difficulties are especially difficult to deal with and greatly slow down the development of lane detection. Due to these difficulties, lane detection remains a challenging vision task.

* Co-corresponding author.

To deal with the first difficulty, many methods characterize lanes with simple fitted curves or masks. For examples, SCNN [Pan et al., 2018] treats the problem as a semantic segmentation task, and introduces slice-by-slice convolutions within feature maps, thus enabling message passing. For these methods, lanes are characterized as a special form (e.g., point, curve or mask), so it is difficult to support the format of marker or grid that usually has an uncertain number. Similarly, those who support the latter [Lee et al., 2017] do not support the former well. To address the second problem, some methods use vanishing point or parallel relation as auxiliary information. For example, a vanishing point prediction task [Lee et al., 2017] is utilized to implicitly embed a geometric context recognition capability. In these methods, they usually only pay attention to a certain kind of structural information or do not directly use it end-to-end, which leads to the structures not fully functioning and the algorithm complicated. For the last problem, some clustering- or detectionbased methods are used to distinguish or classify instances. Line-CNN [Li et al., 2019] utilizes line proposals as references to locate traffic curves, which forces the method to learn the feature of lanes. To these methods, they can distinguish instances and even extend to more attributes, but they usually need extra computation and have many manually designed super-parameters, which leads to poor scalability.

Inspired by these observations and analysis, we propose a novel structure guided framework for lane detection, as shown in Fig. 2. In order to characterize lanes, we propose a box-line based proposal method. In this method, the minimum circumscribed rectangle of the lane is used to distinguish instance, and its center line is used for structured positioning. For the sake of further improving lane detection by utilizing structural information, the vanishing point guided anchoring mechanism is proposed to generate intensive anchors (i.e., as few and accurate anchors as possible). In this mechanism, vanishing point is learned in a segmentation manner and used to produce structural anchors top-down, which can efficiently capture various lanes. Meanwhile, we put forward multi-level structure constraints to improve the perception of lanes. In the process, the pixel-level perception is used to improve lane details with the help of lane binary segmentation, the lane-level relation aims at modeling the parallelism properties of inter-lanes by Inverse Perspective Mapping (IPM) via a neural network, and image-level attention is to attend the image with adaptive weights from the perspective of scenes. Finally, features of lane anchors under structural guidance are extracted for accurate classification, regression and the prediction of other attributes. Experimental results on CULane and Tusimple datasets verify the effectiveness of the proposed method which achieves state-of-theart performance and run efficiently at 117 FPS.

The main contributions of this paper include: 1) we propose a structure guided framework for lane detection, which characterize lanes and can accurately class, locate and restore the shape of unlimited lanes. 2) we introduce a vanishing point guided anchoring mechanism, in which the vanishing point is predicted and used to produce intensive anchors, which can precisely capture lanes. 3) we put forward the multi-level structural constraints, which are used to sense pixel-level unary details, model lane-level pair-wise relation and adaptively attend image-level global information.

2 RelatedWork

In this section, we review the related works that aim to resolve the challenges of lane detection in two aspects.

2.1 Traditional Methods

To solve the problem of lane detection, traditional methods are usually based on hand-crafted features by detecting shapes of markings and fitting the spline. [Veit et al., 2008] presents a comprehensive overview of features used to detect road markings. And [Wu and Ranganathan, 2012] uses Maximally Stable Extremal Regions features and performs the template matching to detect multiple road markings. However, there approaches often fail in unfamiliar conditions.

2.2 Deep Learning basedMethods

With the development of deep learning, methods [Pizzati and Garc´ıa, 2019; Van Gansbeke et al., 2019; Guo et al., 2020] based on deep neural networks achieve progress in lane detection. SCNN [Pan et al., 2018] generalizes traditional deep layer-by-layer convolutions to enable message passing between pixels across rows and columns. ENet-SAD [Hou et al., 2019] presents a knowledge distillation approach, which allows a model to learn from itself without any additional supervision or labels. PolyLaneNet [Tabelini et al., 2020] adopts a polynomial representation for the lane markings, and outputs polynomials via the deep polynomial regression. UltraFast [Qin et al., 2020] treats the process of lane detection as a row-based selecting problem using global features. CurveLanes [Xu et al., 2020] proposes a lane-sensitive architecture search framework to automatically capture both long-ranged coherent and accurate short-range curve information.

In these methods, different lane representations are adopted and some structural information is considered for performance improvement. However, these methods are usually based on the powerful learning ability of neural networks to learn the fitting or shapes of lanes, and the role of scenerelated structural information for lanes has not been paid enough attention to and discussed.

3 The Proposed Approach

To address these difficulties (i.e., characterizing lanes, modeling the relationship between scenes and lanes, and supporting more attributes), we propose a novel structure guided framework for lane detection, denoted as SGNet. In this framework, we first introduce a new lane representation. Then a top-down vanishing point guided anchoring mechanism is proposed, and next multi-level structure constraints is used. Details of the proposed approach are described as follows.

3.1 Representation

For adapting to different styles of lane annotation, we introduce a new box-line based method for lane representation. Firstly, we calculate the minimum circumscribed rectangle R ("box") with the height h and width w for the lane instance Llane. For this rectangle, center line Lcenter ("line") perpendicular to the short side is obtained. And the angle between the positive X-axis and Lcenter in clockwise direction is θ. In this manner, Lcenter provides the position of the lane instance, and h and w restrict the areas involved. Based on R and Lcenter , lane prediction based on points, masks, markers, grids and other formats can be performed. In this paper, the solution based on key points of lane detection is taken just because of the point-based styles of lane annotation in public datasets (e.g., CULane [TuSimple, 2017] and Tusimple [Pan et al., 2018]).

Inspired by existing methods [Li et al., 2019; Chen et al., 2019; Qin et al., 2020], we define key points of the lane instance with equally spaced y coordinates Y = {yi} and yi = H P-1 · i(i = 1, 2, ..., P - 1), where P means the number of all key points through image height, which is fixed on images with same height H and width W. Accordingly, the x coordinates of the lane is expressed as X = {xi}. For the convenience of expression, the straight line equation of Lcenter is defined as

where a, b and c can be easily computed by θ and any point on Lcenter. Next, when the y coordinate of the center line is yi, we can compute the corresponding x coordinate as

Then, we define the offset of x coordinate ∆X between the lane Llane and center line Lcenter as

Therefore, based on Lcenter and ∆X, we can calculate the lane instance Llane. Usually, it is easier to learn Lcenter and ∆X than the directly fitting key points of Llane.

3.2 Feature Extractor

To see Fig. 2, SGNet takes ResNet [He et al., 2016] as the feature extractor, which is modified to remove the last global pooling and fully connected layers for the pixel-level prediction task. Feature extractor has five residual modules for encoding, named as Ei(πi) with parameters πi(i = 1, 2, ..., 5). To obtain larger feature maps, we convolve E5(π5) by a convolutional layer with 256 kernels of 3 × 3 and then ×2 upsample the features, followed by an element-wise summation with E4(π4) to obtain E ′4 (π′4 ). Finally, for a H×W input image, a H 16 × W16 feature map is output by the feature extractor.

3.3 Vanishing Point Guided Anchoring

In order to learn the lane representation, there are two main ways to learn the center line Lcenter and x offset ∆X. The first way is to learn the determined Lcenter directly with angle, number and position regression, which is usually difficult to achieve precise results because of the inherent difficulty of regression tasks. The second way is based on mature detection tasks, using dense anchors to classify, regress and then obtain proposals representing the lane instance. And the second one has been proved to work well in general object detection tasks, so we choose it as our base model.

To learn the center line Lcenter and x offset ∆X well, we propose a novel vanishing point guided anchoring mechanism (named as VPG-Anchoring). The vanishing point (VP) provides strong characterization of geometric scene, representing the end of the road and also the "virtual" point where the lanes intersect in the distance. Since VP is the intersection point of lanes, lanes in the scene must pass through VPs, and lines that do not pass through VPs are not lanes in the scene with high probability. Therefore, dense lines radiated from VPs can theoretically cover all lanes in the image, which is equivalent to reducing the generation space of anchors from RH×W×Nproposal to RNproposal . Nproposal represents the number of anchors generated at one pixel.

As shown in Fig. 2, the features map E ′4 (π′4 ) is feed to VPG-Anchoring. In the mechanism, VP is predicted by a simple branch, which is implemented by a multi-scale context-aware atrous spatial pyramid pooling (ASPP) [Chen et al., 2018] followed by a convolutional layer with 256 kernels of 3 × 3 and a softmax activation. The VP prediction branch is denoted as φV(πV) with parameters πV .

Usually, VP is not annotated in lane datasets, such as CULane [Pan et al., 2018], so we average the intersection points of the center lines of all lane instances and get the approximate VP. In addition, a single point is usually difficult to predict, so we expand the area of VP to a radius of 16 pixels and use segmentation algorithm to predict. To achieve this, we expect the output of φV(πV) to approximate the ground-truth masks of VP (represented as GV) by minimizing the loss

where BCE(·, ·) represents the pixel-level binary crossentropy loss function.

In order to ensure that generated anchors are dense enough, we choose a Wanchor ×Wanchor rectangular area with VP as the center, and take one point every Sanchor to generate anchors. For each point, anchors are generated every Aanchor angle (Aanchor ∈ [0, 180]) as shown in Fig. 4. In this way, anchors are targeted, intensive and not redundant, compared with general full-scale uniform generation and even specially designed methods for lanes [Li et al., 2019]. Note that anchors run through the whole image, and only the part below VP is shown for convenient display in Figs. 2 and 4.

3.4 Classification and Regression

In order to classify and regress the generated anchors, we extract high-level feature maps based on E4(π4) with several convolutional layers. The feature map is named as FA ∈

RH′×W ′×C′ , where H′ ,W ′ and C ′ are the height, width and channel of FA. For each anchor Llane, the channel-level features of each point on anchors are extracted from FA to obtain lane descriptor DA ∈ RH′×C′ , which are used to classify the existence Conf Llane and regress x offsets ∆XLlane including the length len of lanes. To learn these, we expect the output to approximate the ground-truth existence GConf Llane and x offsets G∆XLlane by minimizing the loss

where SL1(·, ·) means smooth L1 loss and L means the number of proposals. Finally, Line-NMS [Li et al., 2019] is used to obtain the finally result with confidence thresholds.

3.5 Multi-level Structure Constraints

In order to further improve lane perception, we ask for the structural relationship between scenes and lanes, and deeply explore the pixel-level, lane-level and image-level structures.

Pixel-level Perception. The top-down VPG-Anchoring mechanism covers the structures and distribution of lanes. At the same time, there is a demand of bottom-up detail perception, which ensures that lane details are restored and described more accurately. For the sake of improving the detail perception, we introduce lane segmentation branch to location lane locations and promote pixel-level unary details. As shown in Fig. 2, the lane segmentation branch has the same input and similar network structure with the VP prediction branch. The lane segmentation branch is denoted as φP(πP) with parameters πP. To segment lanes, we expect the output of PP = φP(πP) to approximate the ground-truth masks of binary lane mask (represented as GP) by minimizing the loss

To promote the pixel-level unary details, we weight the input features FA by the following operation

where MA are feed to classify and regress instead of FA.

Lane-level Relation. In fact, lanes conform to certain rules in the construction process, and the most important one is that the lanes are parallel. Due to imaging reasons, this relationship is no longer maintained after perspective transformation, but it can be modeled potentially. To model the lane-level relation, we conduct IPM by the H Matrix [Neven et al., 2018] via a neural network. After learning H, the lane instance Llane can be transformed to L′lane on bird's eye view, where different instances are parallel. Formally, we define the relationship between lanes as follows. For two lane instances Llane1 and Llane2 in the image, they are projected to the bird's-eye view through the learned H matrix, and the corresponding instance L′lane1 and L′lane2 are obtained. The two instances can be fitted to the following linear equations:

In these two equations, under the condition that y is equal, the difference of x is always constant. Thus we can get that a1 ∗ b2 = a2 ∗ b1. Expanding to all instances, lane-level relation can be formulated as

Image-level Attention. In the process of camera imaging, distant objects are small after projection. Usually, the distant information of lanes is not prominent visually, but they are equally important. After analysis, it is found that the distance between lanes and VP reflects the inverse proportion to scales in imaging. Therefore, we generate perspective attention map PAM based on VP, which is based on the strong assumption that the attention and distance after imaging satisfies two-dimensional gaussian distribution. PAM ensures the attention of different regions by adaptively restricting the classification and regression loss (from Eq. 5) as follows.

where | · | means normalized to [0, 1]. By taking the losses of Eqs.(4),(5),(6),(9) and (10), the overall learning objective can be formulated as follows:

where P is the set of {{πi}5i=1 , π′4 , πV, πC, πR, πP, πL}, and πC, πR and πL are the parameters of classification, regression and lane-level relation subnetworks, respectively.

4 Experiments and Results

4.1 Experimental Setup

Dataset. To evaluate the performance of the proposed method, we conduct experiments on CULane [Pan et al., 2018] and Tusimple [TuSimple, 2017] dataset. CULane dataset has a split with 88,880/9,675/34,680 images for train/val/test and Tusimple dataset is divided into three parts: 3,268/358/2,782 for train/val/test.

Metrics. For CULane, we use F1-measure score as the evaluation metric. Following [Pan et al., 2018], we treat each lane as a line with 30 pixel width and compute the intersection-over-union (IoU) between groundtruths and predictions with a threshold of 0.5 to For Tusimple, the official metric (Accuracy) is used as the evaluation criterion, which evaluates the correction of predicted lane points.

Training and Inference. We use Adam optimization algorithm to train our network end-to-end by optimizing the loss in Eq. (11). In the optimization process, the parameters of feature extractor are initialized by the pre-trained ResNet18/34 model and "poly" learning rate policy are employed for all experiments. The training images are resized to the resolution of 360 × 640 for faster training, and applied affine and flipping. And we train the model for 10 epochs on CULane and 60 epochs on TuSimple. Moreover, we empirically and experimentally set the number of points P = 72, the width of rectangular Wanchor = 40, anchor strides Sanchor = 5 and anchor angle interval Aanchor = 5.

4.2 Comparisons with State-of-the-art Methods

We compare our approach with state-of-the-arts including DeeplabV2 [Chen et al., 2017], SCNN [Pan et al., 2018], FD [Philion, 2019], ENet-SAD [Hou et al., 2019] , PointLane [Chen et al., 2019], RONELD [Chng et al., 2020], PINet [Ko et al., 2020], ERFNet-E2E [Yoo et al., 2020], IntRA-KD [Hou et al., 2020], UltraFast [Qin et al., 2020], CurveLanes [Xu et al., 2020], Cascaded-CNN [Pizzati et al., 2019] and PolyLaneNet [Tabelini et al., 2020].

We compare our approach with 10 state-of-the-art methods on CULane dataset, as listed in Tab. 1. Comparing our ResNet34-based method with others, we can see that the proposed method consistently outperforms other methods across total and almost all categories. For the total dataset, our method is noticeably improved from 74.80% to 77.27% compared with the second best method. Also, it is worth noting that our method is significantly better on Crowd (+2.31%), Arrow (+2.17%) and Night (+3.79%) compared with second best methods, respectively. In addition, we also obviously lower FP on Cross by 3.78% relative to the second best one. As for Curve, we are slightly below the best method (ERFNet-E2E), which conducts special treatment for curve points while maybe damaging other categories. Moreover, our method has a faster FPS than almost all results. These observations present the efficiency and robustness of our proposed method and validate that VPG-Anchoring and multilevel structures are useful for the task of lane detection.

Some examples generated by our approach and other stateof-the-art algorithms are shown in Fig. 5. We can see that lanes can be detected with accurate location and precise shape by the proposed method, even in complex situations. These visualizations indicate that the proposed lane representation has a good characterization of lanes, and also show the superiority of the proposed method.

Moreover, we list the comparisons on Tusimple as shown in Tab. 2. It can be seen that our method is competitive in highway scenes without adjustment, which further proves the effectiveness of structural information for lane detection.

4.3 Ablation Analysis

To validate the effectiveness of different components of the proposed method, we conduct several experiments on CULane to compare the performance variations of our methods.

Effectiveness of VPG-Anchoring. To investigate the effectiveness of the proposed VPG-Anchoring, we conduct ablation experiments and introduce three different models for comparisons. The first setting is only the feature extractor and the subnetwork of classification and regression, which is regarded as "Base" model. In Base, anchor is generated uniformly at all positions of the feature map, and Aanchor is lowered to ensure the same number with SGNet. In addition, we conduct another model ("Base+V") by adding VPG-Anchor. And we also replace the Lcenter by straight line fitted directly by key points as the "Base+V-F" to explore the importance of VP. The comparisons of above models are listed in Tab. 3. We can observe that the VPG-Anchoring greatly improve the performance of Base model, which verifies the effectiveness of this mechanism. In addition, comparing Base+V with Base+V-F, we find the proposed approximate VP in lane presentation is better than the one by direct fitting.

Effectiveness of Multi-level Structures. To explore the effectiveness of the pixel-level, lane-level and image-level structures, we conduct another experiments by combining the pixel-level perception with "Base+V" as "Base+V+P" and adding lane-level relation to "Base+V+P" as "Base+V+P+L". From the last four rows of Tab. 3, we can find that the performance of lane detection can be continuously improved by pixel-, lane- and image-level structures, which validates that the three levels of constrains are compatible with each other, and can be used together to gain performance.

5 Conclusion

In this paper, we rethink the difficulties that hinder the development of lane detection and propose a structure guided framework. In this framework, we introduce a new lane representation to meet the demands of various lane representations. Based on the representation, we propose a novel vanishing point guided anchoring mechanism to generate intensive anchors for efficiently capturing lanes. In addition, multi-level structure constraints is modeled to improve lane perception. Extensive experiments on benchmark datasets validates the effectiveness of the proposed approach with fast inference and shows that the perspective of modeling and utilization of structure information is useful for lane detection.