Real-Time Road Curb and Lane Detection for Autonomous Driving Using LiDAR Point Clouds

The commercialization of automated driving vehicles promotes the development of safer and more efficient autonomous driving technologies including lane marking detection strategy, which is considered to be the most promising feature in environmental perception technology. To reduce the tradeoff between time consumption and detection precision, we propose a real-time lane marking detection method by using LiDAR point clouds directly. A constrained RANSAC algorithm is applied to select the regions of interest and filter the background data. Further, a road curb detection method based on the segment point density is also proposed to classify the road points and curb points. Finally, an adaptive threshold selection method is proposed to identify lane markings. In this investigation, five datasets are collected from different driving conditions that include the straight road, curved road, and uphill, to test the proposed method. The proposed method is evaluated under different performance metrics such as Precision, Recall, Dice, Jaccard as well as the average detection distance and computation time for the five datasets. The quantitative results show the efficiency and feasibility of this proposed method.


I. INTRODUCTION
Recently, self-driving technologies have received much attention due to the several research activities of universityindustry collaboration financed by some reputed companies such as Waymo [1] and Hyundai [2]. However, the realization of autonomous driving technology is difficult. The most basic yet important function for an automated vehicle is environmental perception, because of its direct influence on both drivers and pedestrians. Various objects need to be recognized in the procedure of environmental perception, such as lane markings, pedestrians, and adjacent vehicles. Similarly, different sensing modalities are used to achieve autonomous driving for example, visual and thermal cameras, radars, LiDARs, and ultrasonic sensors. These sensors are equipped alone or combination with other for self-driving vehicles to perceive the driving environment [3]. Among the many targets that need to be perceived, precise lane marking detection is of paramount importance for autonomous vehicles maintaining the advanced driving assistance systems (ADAS), which directly affects the behaviors of driving. Furthermore, the accurate identification of lane markings will be helpful for other transportation applications, such as the localization of vehicles, mobile mapping technology, and optimization of routes. Therefore, lane marking detection is the fundamental feature to develop the autonomous driving system.
Several methods have been proposed to recognize the lane markings from camera images [4], [5]. However, those methods show performance degradation in acquisition of three-dimensional (3D) information of objects, severe weather conditions and non-ideal lighting environments [6]. Radar sensors can work even under adverse weather conditions by directly measuring the radial velocity of objects via the Doppler effect. However, the data collected by radar sensors are noisy [7]. The booming of the selfdriving industry also promotes the commercialization of Light Detection and Ranging (LiDAR) sensors, because of their ability to offer three-dimensional (3D) information of objects. And more importantly, its operation can continue under different weather and illumination conditions. However, the 3D information increases the processing time and complexity due to the tremendous amount of data, especially due to the multiple (16/32/64) laser scanning system. In order to effectively separate the lane marking points from the LiDAR 3D point clouds, road surface points are first extracted from the raw data, then followed by the differentiation of lane marking points from road clouds.
Usually, approaches of lane marking extraction from LiDAR point cloud can be classified into two categories: two-dimensional (2D) image-based methods, and 3D point cloud-based methods. The first step of 2D image-based methods is converting 3D point clouds to 2D images, then, image processing methods are applied to extract road point clouds. Georeferenced intensity images were interpolated from the road points by Guan et al. [8], using an extended inverse distance weighted (IDW) interpolation method, followed by a point-density-based multiple threshold segmentation strategy. Guan et al. [9] generated 2D georeferenced feature images (GFI) from 3D road surface points by using a modified IDW interpolation method. After that, the weighted neighboring difference histogram based dynamic thresholding and multiscale tensor voting algorithms were used to distinguish lane markings from the GFI image. Cheng et al. [10] extracted lane markings from thresholding normalized intensity images by using the deep learning approaches. Ma et al. [11] first segmented road surface from the raw data by using a revised curb-based road extraction method, followed by the IDW approach for 2D georeferenced image generation. Then, a U-shaped capsule-based network was developed to detect lane markings based on the convolutional and deconvolutional capsule strategies. However, there are some apparent shortcomings of the 2D image-based methods. The conversion from 3D point clouds to 2D images requires powerful abilities of computation because of the immense size of data acquired by a multi-array laser scanning system and corresponding post-processing. Furthermore, such conversion from 3D point clouds to 2D images also suffers in high precision loss, which eventually affects the accuracy of measurement for target objects.
The second type of 3D-based method identifies lane markings from the original point cloud directly, without the procedure of synthesizing point clouds to images. Yu et al. [12] proposed a multisegment thresholding and spatial density filtering method to directly extract lane markings after the application of a curb-based road points segmentation approach. Yan et al. [13] first eliminated points in the air and organized useful points to scan lines. Subsequently, road points are extracted by the utilization of height difference (HD) and moving least squares line fitting. In the last step, after the smoothness of intensities by a dynamic window median filter, the edge detection and edge constraint method were operated to identify lane markings.
Jung et al. [14] differentiated drivable regions by set-ting a vertical slope threshold and discriminated lane markings with higher intensities. Here, the marking points are extracted by searching a set of parallel lines with a fixed interval of lane width. However, most of the methods mentioned previously are used for the generation of digital maps and thus not directly focus on autonomous driving. Therefore, the design features of such algorithms are quite different in comparison to actual driving conditions. For example, computational time may not an important factor for the generation of digital map, which it has paramount significance for the application in self-driving. Besides, most proposals are based on the assumption that roads are flat and straight. However, the detection of roads with slopes or curvatures would be challenging in practical scenarios.
In this paper, we present a real-time lane marking detection method for structured roads with few obstructions based on a LiDAR sensor using point clouds. Five datasets were collected from different road conditions including slopes or curvature in shape and investigated thoroughly to verify the feasibility of the proposed method. Overall, the main contributions of this paper are as follows: 1) The spatial distribution characteristics of roads are found, which are used to filter background points.
2) A road curb detection algorithm is proposed to classify curbstone points and road points by using the different distribution features of road and curbstones.
3) An adaptive threshold selection method is proposed to distinguish lane marking points from road points by determining a unique threshold for each scan line.
4) The proposed method is tested on five datasets collected from different road conditions including straight roads, curved roads, flat roads, and slopes.
The structure of this paper is as follows: Section 2 presents the features of devices that are used in this research and a brief overview of five different datasets. Approaches of road data segmentation, the proposed curb points filtering and adaptive threshold selection methods are described in Section 3. Experiments, results, and discussion are concluded in Section 4. Finally, Section 5 summarizes the conclusion.

1) DEVICES
In general, researchers choose to use a mobile laser scanning system [8], [9], 64-beam LiDAR [15], [16], or at least 32beam LiDAR [17], [18] to identify lane markings from point clouds due to its precise 3D information. However, a 16beam commercial LiDAR is used for this research considering the cost-effective application. The five datasets are captured from five different conditions using the RS-LiDAR-16 sensor mounted on a vehicle about 1.5 meters from the ground, as shown in Fig. 1. As Fig. 1, assume that the origin of LiDAR is the origin of data, the driving direction of the vehicle is the positive direction of the X-axis, the left side of driving is the positive direction of the Y-axis, and the vertical direction of XOY plane is the Z-axis. Fig. 1 also shows the commercial Robosense LiDAR sensor that is used for this investigation. This LiDAR comprises 16 laser and detector pairs with a vertical angular resolution of 2°f rom -15°to 15°comprising a total vertical field of view of 30°. The detection range is from 20 centimeters to 150 meters long, with a measurement accuracy within ±2 centimeters. The horizontal field of view is 360°with a data rate of 320000 points per second.

2) DATASETS
In this research, five datasets are used to evaluate the performance of the proposed method in different scenarios, such as the straight road, curved road, and uphill, as presented in Fig. 2. The location of the datasets are in Yubei District, Chongqing, China, where the university campus is situated. Road datasets are collected from inside and outside of the campus area, where each of the road is a couple of kilometers long with diverse scenarios. Dataset 1 is a site of a flat straight road with two white solid lines and a dashed line, while dataset 2 with two white solid lines and a yellow solid line. Dataset 3 is acquired where the road is a straight slope with three white solid lines and one yellow dashed line. The uphill straight road with two white solid lines and a yellow dashed line is also included in this dataset, which is dataset 4. And dataset 5 is a curved and uphill road with two white solid lines and one yellow dashed line. The road slope of dataset 3 is lower than dataset 4 and 5. The exact length of each road and the time of collection are not recorded since we only select 200 frames of data from each dataset. But the number of frames of each of the five datasets are 390, 547, 664, 260, and 239 respectively. This study distinguishes white solid lane markings from 3D point clouds.

III. METHODOLOGY
The workflow of the proposed system is illustrated in Fig. 3. The road surface points are segmented from LiDAR raw data using Random Sample Consensus (RANSAC) algorithm with a plane model [19]. However, there are some constraints are added to this algorithm for this research and will be discussed in a later section. For the road curb points detection, the rough road data points are clustered using the Euclidean clustering algorithm and further used for curb detection algorithm based on segment point density. Finally, the proposed adaptive threshold selection method including refinement strategy is used to identify lane marking points.

A. REGION-OF-INTEREST-SELECTION
This section introduces those approaches that are adopted to select the region-of-interest (ROI). This selective operation will reduce the computational complexity, which eventually helps to reach the goal of real-time lane marking recognition. There are mainly two subsections: road surface data extraction and curb points detection. The segmentation of raw road point clouds eliminates the majority of useless points, whereas the curb detection classifies curb points and road points.

1) EXTRACTION OF ROAD DATA
It is evident that background filtering is crucial for lane marking identification, as the marking points are a small fraction of total point clouds. The RANSAC algorithm is commonly used to extract road data. However, this method fails when there are more points fall on the pedestrian road than driving road [20]. Therefore, we set some constraints on the RANSAC algorithm. Based on the coordinate system as appeared in Fig. 1(a), the passthrough filter with a specific range of one coordinate is set to either remove or preserve surrounding objects in one frame of data. According to the relation between accuracy and distance of target object based on the user manual of RS-LiDAR-16 [21], the accuracy fluctuates after 70 meters, so the range of the X-axis is set from -70 m to +70 m. The threshold of the Y-axis is 10 meters. Then, the original RANSAC algorithm is applied for the ROI selection. This algorithm is reported as the best shape estimation method due to its consideration of slope parameters. First, three points are randomly chosen from point clouds to form a plane model. Then, points that fit this model are retained as inliers, while others are outliers. The number of inliers is recorded and compared between different iterations. In this paper, a distance threshold of 0.07m and iteration number of 200 are experimentally achieved to filter the unwanted points and preserve the useful points as possible. Finally, the best model is determined by the maximum number of inliers.
So far, the rough road data are extracted from raw LiDAR data, where the constrained RANSAC algorithm cannot remove points of road curb. However, the existence of road curb points will influence lane marking detection, as these have reflectivity higher than asphalt points.

2) CURB POINTS DETECTION
Generally, roads include both structured and unstructured data, and lane markings are only present on structured roads. By observing datasets collected in this study, we found that points are arranged in fan-shaped scan lines on the structured road, and the distance between each arc exceeds at least 0.5 meters as shown in Fig. 4(a). Furthermore, roadside data are dense than road data in a certain horizontal range, which is again confirm through data characteristics as shown in Fig.  4(b). Therefore, this statistical feature is used to filter curb points, and the curb detection algorithm based on segment point density is proposed to refine the extracted road points. In order to detect curb points, road points are classified into different clusters based on scan lines. Here, points representing different scan lines are decided by the K dimensional tree (Kd-tree) structure with Euclidean distance threshold dth of 0.2 meters. If the radii of neighbor points in a sphere can satisfy the constraint of, these points are grouped to the same clusters. Later, those clusters are subdivided to several segments by a fixed length of segment LS in the Y direction. However, the real length of a segment RLS is not equal to LS due to the spatial distribution characteristic of points. It is determined by, where, ymax and ymin are the maximum and minimum values of Y within each segment. A more clear explanation of the difference between LS and RLS is indicated in Fig. 5. LS divides the entire cluster into several segments. The density of points within each segment is similar if the segment belongs to the road. On the contrary, if the segment contains the road curb, the point density will increase. RLS is the real length of the segment. This parameter is set because of the discontinuous distribution of points. Points on the road are evenly distributed, so the RLS is quite close to the LS. But for the segment that contains the curb points, points are concentrated at the location of the curb. So the RLS may be smaller than the LS. In conclusion, the LS is only responsible for dividing the data into segments, and the point density within the segment is actually determined by the RLS.
After the division of segments, the number of points within each segment can also be calculated. Therefore, the point density of each segment can be easily achieved using, segmentsize SPD RLS  where, SPD is the segment point density, segmentsize is the number of points within one segment. Now the mean Mean and standard deviation of segments SV can be calculated by, where, NS is the number of segments within each cluster.   6 shows the comparison result of before and after the operation of the curb detection algorithm. Here, the red points in the first row are rough road points. The second row of that figure indicates the results of the road curb points detection algorithm. Points in blue are the recognized road points, while red points are road curb points that will be ignored in the next procedure of lane marking identification.
Therefore, Fig. 5 shows that the points representing roadside curb are successfully filtered, and road surface data are preserved.

B. LANE MARKING IDENTIFICATION AND REFINEMENT
It is quite usual that the reflectivity of white color objects, such as road markings, is higher than in black color, such as asphalt. As the lane markings are made of high reflectivity materials, so the intensity values of lane marking points are higher than asphalt pavement points. Therefore, those markings can be identified using the intensity value of such points. However, intensity information preserved by LiDAR sensors, varies greatly even in the same longitudinal position between different scan lines because of the influence of distance, angle, or road geometry [23], as illustrated in Fig.  7(a). In this figure, X and Y are distance information, Z is the echo intensity. The positive direction of the X-axis is the driving direction, and road points are symmetrically distributed along the Y-axis. There are 6 points with high intensity values for three scan lines are marked as A, B, C, D, E, F. Fig. 7(b) is the bird eye view of Fig. 7(a). It can show the positional relationship of these 6 points more clearly. The y values of points A, B, and C or the other three points D, E, and F are similar. However, the difference in intensity values is quite large. For example, the difference in intensity between A and B is about 20. So, we can conclude that even in the same longitudinal position of different scan lines, the intensity information differs. To solve the above issues, an adaptive threshold selection method is proposed in this paper, and the corresponding results show the successful recognition of lane marking points.
Author Name: Preparation of Papers for IEEE Access (February 2017) VOLUME XX, 2017 6 (a) (b)

FIGURE 7. Intensity distribution: (a) Intensity distribution for three clusters, (b) the bird eye view of the intensity distribution for three clusters.
As mentioned in the previous section, curb points are filtered and pavement points are sorted out based on scan lines after implementing constrained RANSAC algorithm and curb filtering algorithm based on segment point density. Hence, lane marking points can be distinguished from road clouds based on their features within each scan line instead of setting a global threshold [24] or an intensity band [25]. Algorithm 2 shows the detailed description of the proposed adaptive threshold selection approach. In this method, the linearity gray transform is first applied to unify intensity values within each scan line to k: [0, 255] by using (6 Then, calculate probabilities pk for each gray level. Where, num_k is the number of points with gray value k within scan line i. Next, the mean value ave0 is set as the initial threshold, and the threshold interval is defined as [ave0, 255]. Therefore, the optimum threshold is determined by the equations (11) where th is the current threshold, and points in scan line i with gray values higher than ave0 will be divided into two classes. Points with gray values higher than ave0 but smaller than th are class 1, while points with gray values belong to the range [th, 255] are class 2. So, ω1i is the probability of class 1 for scan line i. μ1i is the mean gray value of class 1 for scan line i. The following equation shows the inter-class variance.
In this equation, σi 2 is the inter-class variance of scan line i, and μGi is the global mean. ω2i is the probability of class 2, and μ2i is the mean gray value of class 2. They use the same calculation equations as like class 1, where only the range of k is different.
The desired threshold is the value of th that maximizes the inter-class variance. Compared with the original Otsu's method [24] that determines the best possible threshold by stepping through integers from 0 to 255 as potential thresholds, this method reduces the number of calculations by about 200 for one scan line by using the redefined threshold selection interval. Besides, the best threshold determined by this method is more appropriate because of the reduction of interference from road points with lower gray values.
In addition, to further optimize the results of the adaptive threshold selection method, the RANSAC algorithm with a line model is performed to filter noises far away from lane markings in the driving direction. This approach is implemented based on the assumption that lane markings are organized regularly in the shape of straight or curved. The reason for using model-based method is that it has good robustness when the lane markings are worn or blocked by obstacles [26]- [28]. Fig. 8 shows the outcomes of the ROI selection approach that includes the results of the rough road data with marks of curb points as well as, the results using the curb filtering algorithm based on segment point density. Fig. 8(a) shows a frame of the original point clouds. The extraction of road surface data and its effect is displayed in Fig. 8(b). After operating the constraint RANSAC-plane algorithm, rough ground data are segmented from the whole point clouds. Here, the blue points are rough road data, while other objects are in red. It is clear that some curb points still exist as the result of road segmentation as shown in Fig. 8(c). The selected area is enlarged in a yellow box, where the circles indicate the positions of road curb points. Subsequently, the curb detection algorithm based on segment point density is carried out, and results are presented in Fig. 8(d). At this stage, the ROI selection of point clouds is completed, and ground points are well prepared for further processing.

B. QUALITATIVE RESULTS
The application of the adaptive threshold selection method and its results based on five datasets are presented in Fig. 9. Green, blue and red points are identified as lane markings, road and other objects, respectively. Fig. 9(a) is a sample of dataset 1, which is collected from an ideal structured road. It is a double driving straight road without any slope. Fig. 9(b) is a sample from dataset 2 which is also collected from a flat double-driving straight road. The only difference is that there is a dashed white line in dataset 1 but a yellow solid line in dataset 2. Some parts of the white dashed line are in green. Due to the long gap between each scan line with data processing frame by frame, dashed lines cannot be fully recognized. Here, the two white solid lines are identified. Fig.  9(c) is the result of dataset 3, which is collected from an incline roadside parking lot with three white solid lines, which are successfully detected. A result of an uphill road with two solid lines is drawn in Fig. 9(d). In Fig. 9(e), the sample of an uphill curved road with two solid lane markings is presented. The yellow box marks a moving vehicle which shows that the proposed method can determine the lane line

C. QUANTITATIVE RESULTS
To evaluate the performance of the proposed lane marking detection algorithm, different performance metrics are investigated [29]. Here, four coefficients are commonly utilized: Precision, Recall, Dice, and Jaccard.  Results of Dice and Jaccard confirm that the proposed method outperforms the results as appeared in [29], where a similar 16-beam-LiDAR has been used for investigation.
The comparison of dataset 1 and dataset 2 shows that white lines can be detected, even some parts of the dashed line. The results of dataset 3 clarify that a little slope will enhance the detection quality of lane markings, compared with the flat road in dataset 1. But if the slope is too high, as the uphill in dataset 4, the quality can be affected. The comparison of results of dataset 5 and dataset 4 shows that the curvature of roads does influence the quality of detection. Fig. 10 to Fig. 14 present the histogram of Precision, Recall, Dice and Jaccard coefficients for the respective five datasets. The occurrence frequency is normalized so that the sum of heights of all bars equals 1.      Table 2 lists the average detection distance of the adaptive threshold selection method for 200 frames data from every dataset. Since this method process data frame by frame, the detection range shown in Table 2 is the average distance per frame. The average detection distance for the five datasets is 47.14 m, 33.52 m, 42.41m, 33.01 m, and 38.31 m, respectively. The curb filtering and lane marking identification algorithm proposed in this research is written in C++ and operated on an Intel Core i5-1035G7 computer in the Linux system. The computation time of five datasets for extracting rough road points and detecting lane marking points is stated in Table 3. The time used for lane marking detection for the five datasets is 38.70 ms, 29.07 ms, 31.50 ms, 32.17 ms, and 25.07 ms, respectively. The total running time is no more than 50 ms per frame. This fast response time is quite effective for lane marking detection especially using LiDAR sensors in autonomous driving. As stated before, many researchers used LiDAR to develop digital road maps, and thus the computational time is not a primary factor to consider for their investigation. According to the regulations of speed limitation, the maximum speed on the expressway cannot exceed 120 km/h, which is 33.3 m/s. Therefore, the proposed method could detect lane markings reach at least 33 m within 50 ms. Hence, the presented results justify the use of the proposed method for the detection of lane marking in real-time.
However, there are still some opportunities to improve the presented results such as the full identification of dashed lines and yellow lines. The quantitative evaluation results still need to be improved. After all, safety is of paramount importance. Besides, other markings play a significant role in guiding the behavior of vehicles as well. Our future direction will focus on solving these issues.

V. CONCLUSION
An efficient method with real-time operation is proposed in this research to filter curb points and detect lane markings on the structured roads from 3D point clouds operated by costeffective 16-beam LiDAR. The overall system includes two sections: ROI selection and lane marking identification. In the ROI selection, a constrained RANSAC algorithm, and the newly designed curb detection approach based on segment point density were applied to achieve the road data segmentation and refinement. Additionally, in the lane marking recognition part, an adaptive threshold selection method was implemented to distinguish marking points from ground point clouds and reduce the computation time. The proposed method was tested for five datasets with diverse road conditions as prepared by our research laboratory. Moreover, the quantitative results are evaluated by performance metrics like Precision, Recall, Dice, and Jaccard. Here, the average results for five datasets are 94.49%, 87.37%, 90.40% and 82.82%, respectively. Also, the average detection range for a frame is 47.14 m, 33.52 m, 42.41 m, 33.01 m, and 38.31 m, respectively. In addition, the average computation time is no more than 50 milliseconds per frame, which eventually shows the efficient operation on a real-time basis of the proposed method.