Digital Commons @ Michigan Tech Digital Commons @ Michigan Tech A comparison of two open source LiDAR surface classification A comparison of two open source LiDAR surface classification algorithms algorithms

: With the progression of LiDAR (Light Detection and Ranging) towards a mainstream resource management tool, it has become necessary to understand how best to process and analyze the data. While most ground surface identification algorithms remain proprietary and have high purchase costs; a few are openly available, free to use, and are supported by published results. Two of the latter are the multiscale curvature classification and the Boise Center Aerospace Laboratory LiDAR (BCAL) algorithms. This study investigated the accuracy of these two algorithms (and a combination of the two) to create a digital terrain model from a raw LiDAR point cloud in a semi-arid landscape. Accuracy of each algorithm was assessed via comparison with >7,000 high precision survey points stratified across six different cover types. The overall performance of both algorithms differed by only 2%; however, within specific cover types significant differences were observed in accuracy. The results highlight the accuracy of both algorithms across a variety of vegetation types, and ultimately suggest specific scenarios where one approach may outperform the other. Each algorithm produced similar results except in the ceanothus and conifer cover types where BCAL produced lower errors.

Abstract: With the progression of LiDAR (Light Detection and Ranging) towards a mainstream resource management tool, it has become necessary to understand how best to process and analyze the data. While most ground surface identification algorithms remain proprietary and have high purchase costs; a few are openly available, free to use, and are supported by published results. Two of the latter are the multiscale curvature classification and the Boise Center Aerospace Laboratory LiDAR (BCAL) algorithms. This study investigated the accuracy of these two algorithms (and a combination of the two) to create a digital terrain model from a raw LiDAR point cloud in a semi-arid landscape. Accuracy

OPEN ACCESS
of each algorithm was assessed via comparison with >7,000 high precision survey points stratified across six different cover types. The overall performance of both algorithms differed by only 2%; however, within specific cover types significant differences were observed in accuracy. The results highlight the accuracy of both algorithms across a variety of vegetation types, and ultimately suggest specific scenarios where one approach may outperform the other. Each algorithm produced similar results except in the ceanothus and conifer cover types where BCAL produced lower errors.

Introduction
Developing accurate Digital Terrain Models (DTM) has been a long stated goal of both researchers and resource managers interested in quantifying land surface elevations. The potential applications of a reliable DTM include habitat assessment, forest succession, snowmelt simulation, hydrologic modeling, carbon sequestration, glacial monitoring, and floodplain assessments [1][2][3][4][5][6]. Prior to the introduction of Light Detection and Ranging (LiDAR), traditional methods such as photogrammetry and field surveys were conducted to produce DTMs. While these methods can generate DTMs with acceptable levels of accuracy for certain applications, both methods are time and labor intensive. Furthermore, in the presence of steep slopes or high biomass, traditional DTM generation methods are difficult to implement, often leading to reduced levels of accuracy [7][8][9]. Research has demonstrated that LiDAR DTM generation is more efficient and accurate as compared to traditional methods [10]. The creation of accurate DTMs is vital to understand the reliability of other LiDAR-derived metrics, such as canopy cover, tree heights, and leaf area index [11].
In recent years investigations have focused on the influence of environmental conditions (e.g., slope, elevation, cover type) and sensor characteristics (e.g., flight height, point density, and scan angles) on the accuracy of LiDAR-derived DTMs [2,[12][13][14][15][16][17][18][19], ultimately demonstrating the reliability of LiDAR-derived DTMs across a range of terrain and cover types with varying acquisition parameters. Other studies investigated the impact that different point interpolators have on the accuracy of LiDAR-derived DTMs [20,21]. Most suggest that when the LiDAR acquisition has an adequate pulse density, usually equal to, or denser than the desired DTM resolution (cell size), there are only negligible differences in accuracy due to interpolation methods. These past studies provided insight into the reliability of LiDAR to create accurate DTMs in various landscape types and have developed guidelines for working with LiDAR in different ecosystems [22].
Raw LiDAR point clouds contain returns from both ground and non-ground objects. Classification of these points as ground or non-ground returns is the first step in generating a DTM from LiDAR data. Although the influence of different variables like pulse density, terrain slope, and vegetation on the vertical accuracy of LiDAR-derived DTMs has been assessed, little attention has been paid to the accuracies of the different point classification algorithms commonly being applied. Point classification algorithms that are commonly applied by LiDAR vendors are considered proprietary knowledge, are often grey-or black-box approaches, and thus are not readily available for independent validation and comparison. For landowners that will only every acquire LiDAR once or twice, the use of these proprietary methodologies is limited by the high cost of purchasing the software ($5,000-$20,000). However, in recent years open source point classification algorithms have been developed. Since they are open source, such algorithms can be independently tested, evaluated, and compared against the products commonly produced by vendors. Two such algorithms are the Multiscale Curvature Classification LiDAR algorithm (MCC, http://sourceforge.net/project/mccLiDAR) and the Boise Center Aerospace Laboratory LiDAR algorithm (BCAL).
The MCC algorithm was developed at the Moscow Forestry Sciences Laboratory of the USFS Rocky Mountain Research Station [23] and the BCAL algorithm was developed by the Boise Center Aerospace Laboratory of Idaho State University [24]. The two algorithms were developed for different objectives; MCC was intended for classifying LiDAR returns in high biomass forest ecosystems [2,[25][26][27][28], while the BCAL algorithm was developed specifically for optimal performance in shrub-steppe ecosystems [19,24,29,30]. The primary difference between the way the methods iteratively interpolate surfaces to the point cloud during processing is that MCC works from the top down while BCAL works from the bottom up. The MCC algorithm operates by discarding returns that exceed a threshold curvature calculated from a surface interpolated using a thin plated spline. Through three successively larger scale domains that define the processing window size, the algorithm iterates until the number of remaining returns changes by <1%, <0.1% and finally <0.01% for the three scale domains, respectively [23]. BCAL is a grid based classification algorithm, first identifying the lowest elevation point in a search area determined by the user, and then creating a surface by interpolating these lowest points [24]. In subsequent iterations, any point that lies on or below the previous iteration's surface are classified as ground points and are included in subsequent iterations. The iterations continue until there are no unclassified returns below the interpolated surface in which case all unclassified returns above the surface are classified as vegetation returns.
The objective of this study is to cross compare and evaluated the ability of the two algorithms to filter a raw LiDAR point cloud and produce an accurate DTM in a semi-arid watershed with mixed vegetation types. The performance of each algorithm is tested for overall accuracy, in a variety of cover types and at different spatial resolutions.

Study Area
The study area is the 38 ha Reynolds Mountain East (RME) catchment, which is part of the larger Reynolds Creek Experimental Watershed ( Figure 1). Located in southwestern Idaho, USA, the watershed is owned by the Bureau of Land Management and research infrastructure in the watershed is managed by the USDA ARS Northwest Watershed Research Center. The catchment is in a semi-arid mountainous area ranging from 2,023 to 2,139 m in elevation with slopes reaching 35°. The landscape is a patchwork of shrub-steppe, meadow, and bare ground with large stands of coniferous and deciduous forests that occur in topographically sheltered zones. The shrub-steppe in RME is dominated by mountain big sagebrush (Artimesia tridentate Nutt. ssp. vaseyana) and mountain snowberry (Symphoricarpos oreophilus Gray) with patches of prostrate ceanothus (Ceanothus prostratus Benth.). The meadow is a mixture of forbs and graminoids with Lupinus ssp. and Carex ssp.
dominating the forbs and Poa ssp. dominating the graminoids. The conifer tree stand is solely comprised of Douglas-fir (Pseudotsuga menziesii), while quaking aspen (Populus tremuloides) dominates the deciduous stand.

Figure 1. Cover type map of Reynolds
Creek Mountain East, defined by seven classes (Willow, Ceanothus, Shrub, Bare Ground, Conifer, Meadow, and Aspen). The shrub cover type represents all areas not outlined by other cover types and the willow cover type was excluded from surveying due to the impracticality of operating equipment in saturated areas. Each set of yellow points represents the location of a 4 m by 4 m plot with survey points systematically gridded every 0.5 m (81 points/plot; 9 points/m 2 ; see inset).

LiDAR Data and Acquisition
Airborne LiDAR data were acquired in mid-November 2007 with a Leica ALS50 Phase II Laser, which operated at a wavelength of 1,064 nm and records up to four returns per pulse. The data were acquired with a nominal pulse density of 6 pulses/m 2 and an off nadir scan angle of ±15°. The mean return rate and range of return rates across the study area were 5.7 returns/m 2 and 0-70 returns/m 2 , respectively. According to vendor provided measurements, the absolute vertical accuracy in terms of root mean square error (RMSE) was 3.3 cm based on 1,002 real time kinematic (RTK) global positioning system (GPS) points surveyed on asphalt road surfaces throughout the watershed. The flight lines were calibrated and the raw LiDAR data were tiled using the TerraScan and TerraMatch software respectively (Terrasolid Ltd., Jyvä skylä , Finland), the delivered raw bins were projected in NAD83 UTM Zone 11 North. Along with the xyz coordinates, GPS time, intensity, and scan angle were also recorded for each return. A study using the same LiDAR dataset, from another area of the watershed, found a vertical accuracy of approximately 10 cm and horizontal accuracy within 30 cm [29].

Point Classification with MCC and BCAL
The raw LiDAR point cloud was classified into ground and non-ground returns with both of the previously described algorithms (MCC and BCAL). Ranges for initial parameters were selected for both methods by consulting with the algorithm developers, for MCC the scale and curvature ranged from 0.8 to 1.5 and 0.01-0.10, respectively, and for BCAL the window and threshold ranged from 5 to 7 m and 0.00-0.10 m, respectively. These initial parameters were used to produce DTMs for an optimization process in order to determine which parameter settings produce the lowest root mean square error (RMSE) for the study area. We acknowledge that a similar optimization process will be necessary for other study areas, especially with different vegetation types such as forests. The MCC parameters found to operate the best were a scale value of 1.0 and curvature value of 0.05. Both the original BCAL algorithm and a modified version were applied to the raw LiDAR data set. In the modified BCAL algorithm, a threshold level is applied, which instead of classifying LiDAR returns using the absolute interpolated surface, the interpolated surface plus a threshold value of 0.05-0.10 m was used at each iteration for classifying LiDAR data into ground and non-ground returns. The threshold was applied to allow for an increase in the proportion of returns being classified as ground returns. Through the parameterization process the modified BCAL algorithm was found to outperform the original at all tested resolution levels. The modified algorithm produced the lowest RMSE when the window size was set to 7 m and the threshold value was 0.10 m. Then combinations of the MCC and the modified BCAL algorithms were also applied for filtering the raw point cloud, it was found that using the BCAL window size of 7 m and threshold of 0.10 m with the MCC scale value of 1.0 and a curvature value of 0.05, produced the lowest RMSE of the combined filtering results. This new point classification method will be referred to as Combo through the rest of the paper. The algorithm parameters that are utilized in the analysis are summarized in Table 1.

Ground Reference Data
In summer 2010, 71 plot locations were proportionally allocated by area, within six vegetation cover type strata (shrub, ceanothus, meadow, bare ground, deciduous, and conifer; Figure 1). The sampling protocol ensured that the selected plots covered the range of terrain and vegetation variability within the study area (Table 2). Each plot consisted of a 4 × 4 m square within which survey points were systematically distributed on a 0.5 m grid spacing (81 survey points per plot). Each survey point was located and measured (x, y, z position) using a Topcon GTS-236w laser total station that had been georeferenced using a Topcon Hyper-Pro real-time kinematic (RTK) global positioning system (Topcon Corp., Livermore, CA, USA) and the system of USGS monuments (Figure 1). The Topcon Hyper-Pro RTK has a static reading standard of ±15 mm vertical and ±10 mm horizontal. At each of the 81 survey points within the grid a presence/absence of vegetative cover was recorded for calculating the average vegetation cover of the plot (Table 2). In addition to the 81 survey points, additional survey measurements were taken at the center of each shrub (center of crown area) and on the northern side of each tree bole in those plots with trees. This provided a total of 7,285 points for assessing the accuracies of the DTMs. Once the LiDAR point cloud had been classified into ground and non-ground returns with the different combinations of the algorithms and parameters, DTMs were generated from the ground returns via a natural neighbors interpolator. DTMs were generated at 1.0 and 0.5 m cell size resolutions for each of the three algorithms (MCC, BCAL, Combo). It is believed that with the density of returns in this LiDAR data set that error introduced through interpolation is negligible no matter what interpolator is being used [20,21]. The survey point locations were used to extract the corresponding DTM elevations for comparison. The corresponding surveyed elevations were then subtracted from the DTM elevations and these values were analyzed using the Kruskal-Wallis one-way ANOVA. Utilizing the ANOVA, all 7,285 survey points were used to assess the overall performance of the DTM in terms of RMSE across the catchment. Furthermore, the DTMs were analyzed to evaluate which method performed the best within each of the cover types. All results are reported at a significance level of p < 0.05.

1.0 m Resolution
When assessing the overall performance of the algorithms, BCAL, MCC and the Combo returned RMSEs of 27.33, 27.98, and 30.41 cm, respectively. The ANOVA indicated that there was no significant difference between BCAL and MCC, but that each outperformed the Combo algorithm. We further compared the performance of the algorithms within each of the cover types ( Figure 2) and observed that BCAL was either comparable to (or significantly better than) the other two methods in each of the cover types. At 1 m resolution, BCAL provided the most consistent performance between the cover types; however, the difference in RMSE between BCAL and MCC overall was only 0.65 cm, an increase of 2.4%. The ceanothus class was the only one where a substantial deviation (~40 cm) occurs between the BCAL and MCC DTMs, it appeared that MCC misclassified several vegetation returns. In other areas, however, it was evident that BCAL removed ground returns from rock outcroppings that MCC had done a better job of retaining, this misclassification caused BCAL's DTM surface to be nearly 2 m below actual.  0 m (a) and 0.5 m (b) were tested to identify the best overall performance in each of the cover types. At 1.0 m resolution there were no significant differences reported in the shrub, forb and bare ground cover types, nor were there significant differences within the shrub and bare ground cover types at 0.5 m resolution. Significance was tested at the p = 0.05 level.

0.5 m Resolution
At 0.5 m resolution, BCAL, MCC and the Combo returned overall RMSE levels of 27.18, 27.65, and 28.90 cm, respectively. Again the ANOVA indicated no significant difference between BCAL and MCC, and that both slightly outperformed the Combo algorithm. When testing the algorithms within the different cover types, BCAL outperformed both MCC and the Combo in all of the cover types except the bare ground and shrub classes (Figure 2). At 0.5 m resolution, BCAL provided only marginally better accuracy, with a 0.47 cm or 1.75% increase in RMSE between BCAL and MCC.
These two algorithms were further investigated by comparing the MCC and BCAL DTMs, at the 0.5 m cell size, with a surface created from the grid of plot points, to show the differences between specific scenarios (Figure 3). Image (a) of Figure 3 indicates the challenge of filtering rock outcrops. In this case, when subtracting the survey produced surface from the algorithm surfaces, BCAL had a mean and standard deviation of −92 and 74 cm, while MCC had a mean and standard deviation of −61 and 46 cm. BCAL removed returns that fell on rock outcrops, causing the DTM surface to be more than 2 m below the 'Truth Surface'. Image (b) demonstrates how the methods perform in the dense vegetation of the ceanothus. In this case, BCAL had a mean and standard deviation of −17 and 5 cm, while MCC had a mean and standard deviation of 20 and 13 cm. Within the ceanothus class it appears that MCC failed to filter much of the vegetation returns from the ground returns. Overall the performance of BCAL and MCC produce comparable DTMs, although they differ in ways that reflect their contrasting processing approaches. MCC caused commission errors with its misclassification of some vegetation returns as ground, while BCAL produced omission errors by omitting some actual terrain features.

Conclusions
This study assessed the potential of the BCAL and MCC algorithms to perform in a mixed cover type landscape with a variety of terrain features. Ideally, multiple algorithms could be applied to different cover types if a highly accurate DTM is required. Table 3 summarizes the performance of each algorithm within the different cover types, these results should be used in deciding which processing method will work the best within a LiDAR acquisition. It can be seen that both BCAL and MCC performed well overall, but BCAL performed better within the ceanothus and conifer cover types. It should be noted that both algorithms tended to produce surfaces with negative elevation biases when compared with the survey points, meaning that predicted elevations were less than those observed. This trend was consistent for all cover types except the ceanothus as predicted by MCC, here there is a large positive bias due to MCC's failure to classify points within this dense mat of vegetation. The points returned by vegetation like ceanothus (Figure 3(b)) create a plainer feature that MCC is unable to filter, while the block minimum approach that BCAL utilizes is able to remove most of these features.
Separate assessments at both the 1.0 m and 0.5 m resolution level revealed no significant difference between the BCAL and MCC accuracy levels, while both algorithms outperformed the Combo algorithm. The DTM results were also compared between the 1.0 m and 0.5 m resolutions; again this showed no significant difference in accuracy for any of the algorithms/methods. However, it was observed in the DTMs that Combo did experience the greatest improvement in accuracy in each of the cover types and overall, when going from 1.0 m to 0.5 m resolution. The improvement is attributed to the higher percentage of points that MCC retains during classification. Both algorithms have potential to be applied to most cover types but one may outperform the other in specific scenarios. The results indicate that BCAL is able to create a more reliable surface in very dense, continuous vegetation like that which occurs in the ceanothus and willow cover types. This is most likely due to the block minimum approach that BCAL uses, allowing it to create a surface from fewer points than MCC. In areas where steep or sudden changes in slope, such as with the rock outcroppings, MCC is expected to outperform BCAL because it retains more ground returns during the filtering process. Although the ANOVA indicates that BCAL and MCC will provide similar overall accuracies when optimally parameterized, a prior knowledge of the terrain and vegetation within a LiDAR acquisition area, and the particular project objective(s), should be considered when deciding which algorithm to employ. Further analysis will be necessary to fully understand how versatile these algorithms are or if they can truly be applied to all landscapes.