Road Segmentation and Environment Labeling for Autonomous Vehicles

Abstract

In autonomous vehicles (AVs), LiDAR point cloud data are an important source to identify various obstacles present in the environment. The labeling techniques that are currently available are based on pixel-wise segmentation and bounding boxes to detect each object on the road. However, the Avs’ decision on motion control and trajectory path planning depends on the interaction among the objects on the road. The ability of the Avs to understand the moving and non-moving objects is the key to scene understanding. This paper presents a novel labeling method to combine moving and non-moving objects. This labeling technique is named relational labeling. Autoencoders are used to reduce the dimensionality of the data. A K-means model provides pseudo labels by clustering the data in the latent space. Each pseudo label is then converted into unary and binary relational labels. These relational labels are used in the supervised learning methods for labeling and segmenting the LiDAR point cloud data. A backpropagation network (BPN), along with traditional gradient descent-based learning methods, are used for labeling the data. Our study evaluated the labeling accuracy of two as well as three layers of BPN. The accuracy of the two-layer BPN model was found to be better than the three-layer BPN model. According to the experiments, our model showed competitive accuracy of 75% compared to the weakly supervised techniques in a similar area of study, i.e., the accuracy for S3DIS (Area 5) is 48.0%.

1. Introduction

Autonomous vehicles rely on machine learning algorithms [1] and efficient computational software to analyze data from various sensors to understand surrounding obstacles to navigate on the road. Each type of sensor provides a different dataset, suitable for understanding the surrounding environment and vehicle movement on the road. Video cameras offer streaming video and signal information. LiDAR sensors provide better-defined images, even under unfavorable weather conditions. The bus data comprise acceleration, velocity, GPS coordinates, brake pressure, pitch, and roll angles; blending all these signals achieves high reliable data suitable for determining the distance of the obstacles from the autonomous vehicle’s view. Autonomous vehicles require many human labels for detecting the objects in the scene and eventually use these labels for supervised learning techniques. Consequently, the autonomous vehicle industry spends considerable capital to collect human-labeled datasets in different environmental conditions. Improving the systems’ efficiency in detecting objects in various geographical conditions involves collecting new labels with 3D bounding boxes.

In recent years, many datasets have been collected in various cities to solve the different comprehensive visual navigation tasks, such as instance segmentation, object detection, 3D geometry estimation, and localization [2]. The most comparable datasets [3,4] focusing on multimodal sensors are KITTI, Lyft Level 5, nuScenes, Mapillary Vistas, Cityscapes, ApolloScape, Waymo OD, A2D2, and Euro-PVI. These datasets contain LiDAR 3D point clouds and images collected from various sensors. Among these, A2D2 [4] emphasizes more LiDAR data and bus signals for segmentation. The A2D2 dataset is used in our research to segment and label 3D point cloud data.

Object detection and segmentation of road scenes using 3D point cloud data is a substantial problem in computer vision. Riegler et al. [5] proposed an adaptive space partitioning scheme that focuses computations on data structures on 3D voxel grids. The voxel-graph network proposed in [6] learns the feature representation by constructing the KNN graph through all voxels. Octrees are one of the most popular techniques on voxel grids to learn the representations of high-resolution 3D data. PointTrackNet [7] tracks the object displacement point-wise for any detected object. The network is evaluated using the KITTI tracking dataset for various scenarios by processing the adjacent data frames.

Point cloud labeling [8] is an essential step in the segmentation process, where each point is assigned a class, and later it serves as a cue in target detection and scene segmentation. Supervised learning algorithms require numerous labeled data and more computational power due to the high dimensional data. The whole process of manual labeling is time, effort and cost intensive, as it involves a lot of human effort [9]. As an alternative, weakly supervised learning techniques segment the scenes with fewer labels. Xu and Lee [10] proposed weakly supervised learning for point cloud segmentation of 3D CAD shapes and indoor scenes by identifying spatially adjacent points with similar colors and assuming that point cloud prediction is rotation and view angle invariant. However, in reality, the segmentation algorithms for road scenes should handle multiple frames from different sensors, capturing objects in multiple view angles around the AVs.

In [11], the authors proposed an unsupervised 3D object detection module for leveraging the underlying 3D geometric structure of the point cloud and integrated it with the knowledge transfer network pre-trained with the existing images. Generally, most systems perform the object classification algorithm using color images, such as pixel-prototype contrastive learning [12]. However, LiDAR provides road information as first-hand data when the self-driving system detects surrounding objects. The latency in a streaming detection system should be improved for object detection, as given in [3]. Therefore, applying the object classification algorithm directly to the point cloud data will achieve a faster judgment in path planning. There are only a few weakly supervised segmentation methods. The approach in [13,14,15] tried to capture the geometric structure of the point cloud without converting them into depth maps.

The current state-of-art methods for the weakly supervised environment segmentation, classification, and object localization aim to recognize objects individually and learn the data patterns for each group of similar class objects. However, they neglect to visualize the relationship between different class objects. For example, they did not analyze the relational patterns by grouping moving and non-moving objects together, for example, a vehicle adjacent to the divider or a car adjacent to the vegetation. As far as our knowledge, very few group-wise weak supervised algorithms are available in the industry for image and point cloud data segmentation. Zhou et al. [16] proposed group-wise learning to understand the relationship between two images by capturing the semantic context and underlying visual patterns. In this work, a graph neural network conducts multiple iterations to discover semantic information and meaningful relationships among a group of images but does not learn about the relationship between different object classes within the same image. In [17], the authors proposed a 3D object-object relationship graph model using geometric appearance and spatial features of point cloud data. The authors learned about the relationship between different objects but the extracted patterns are specific to each object class for the 3D bounding box proposal using supervised learning.

The onboard perception system would require a database with different group-wise label categories based on relational attributes, such as moving and non-moving objects to deploy autonomous vehicles safely. Keeping in mind the limitations of the previous unsupervised learning approaches for group-wise or relational learning for 3D structures, we have conducted a pioneer attempt to introduce relational labels for road scene segmentation. Autoencoders combined with K-means are a less expensive way to cluster the unlabeled data. They lead to performance gains across various architectures and labeling strategies. Furthermore, the relational labeling technique using unsupervised learning is combined with supervised learning methods in order to improve the accuracy and refine the model predictions for the segmentation task.

The main contributions of the present research study are as follows:

• In the study, an object labeler operates directly from bird’s eye view (BEV) LiDAR point cloud data and reduces the computational burden by processing only the non-ground data.
• The human-labeled dataset is not sufficient for autonomous vehicles. It is crucial to enhance the label in order to improve detection and segmentation accuracy in real-time. The research introduces unsupervised techniques to create pseudo-labels combined with a supervised technique to build an automated labeling system to solve the problem.
• Our method can quickly encode point clouds by learning the relationship between two objects, obtaining the spatial feature of point clouds, labeling multiple categories, and predicting various categories. These labels can help train the network and perform 3D segmentation in real-time.

2. Proposed Method

In recent years, deep learning-based computer vision technologies have been a hot topic for research in the feature extraction of 3D LiDAR point cloud data for autonomous vehicles. It is beneficial for determining 3D object contours and their labels in the 3D space. It has a pivotal role in understanding the scenes and has many applications in various fields, such as 3D medical imaging, robotics, terrestrial mapping, virtual reality, and many more. Before developing deep learning models, the feature representation used the histogram of oriented gradients (HOG), rotation HOG descriptors, and harmonic descriptors’ for showing the 3D shapes. In autonomous vehicles, 3D LiDAR point cloud data combined with deep learning models [5] are applied in 3D object recognition and 3D semantic segmentation.

2.1. Feature Learning and Preprocessing

Feature learning is a part of the preprocessing step for understanding the dataset before it can be applied to train the network for object detection, semantic segmentation, and classification tasks. Feature learning [6] of the point cloud data can be challenging, as it is very unorganized. The multidimensional point cloud data take more computational time to classify the object on the road. In the case of road scenes, targets such as vehicles, buildings, and vegetation have varied heights compared to the flat road surface. The reflectance value of these objects is much higher than the road surfaces. Thus, filtering out the point cloud based on the height and reflectance value is the preliminary criteria for separating the non-ground points from the ground points [18].

Recent studies have shown that separating ground and non-ground points makes the segmentation easier for extracting the region of interest (ROI). Douillard et al. [19] proposed various techniques to identify the ground points and have used them as data separators to segment the objects in the road scene. Zhou et al. [20] proposed a CNN-based method to learn features for scene and object recognition tasks. The neural network uses the scene-centric database with over 7 million labeled pictures of scenes called places to train the model and establish a new state-of-the-art technology for learning the diversity of the scene objects.

Object detection and segmentation tasks in many real-world applications require multi-instance and multiview labeled training samples. In most cases, the data are weakly labeled or unlabeled for changing and unpredictable environments. Many machine learning techniques have been studied to train the classifier with unlabeled training samples. Xu et al. [21] proposed a co-labeling technique to classify the view from a set of pseudo labels generated by training another view.

2.2. Autoencoders

Autoencoders (AE) are categorized into many types, and their use varies, but perhaps the more common use involves feature learning or representation learning [22] for clustering similar points together. Feature learning [23] is significant for segmentation of the road scenes; among multiple technologies [24,25,26,27] prevailing in the community, the unsupervised learning technology such as AE [28] has a wide range of applications, especially in clustering nonlinear subspaces. Autoencoders are also used to compress point clouds [29] and can reconstruct high-quality point clouds from the latent space.

Training the point cloud using the deep learning model [30,31] poses great challenges, due to its high dimensionality and non-linearity. In [32], the authors proposed fully connected autoencoders, which directly consume point clouds with a pre-processing operation and are less prone to permutation variance in the point clouds. This architecture was extended to PointNet++ [33], where the authors used the previous architecture to recursively partition the input point cloud to capture the local structures in a metric space, which was not achieved in the previous model.

Yang et al. [34] proposed autoencoders that successfully extract localized features for generative networks. The generative models [35] are trained in the latent space and achieve greater reconstruction accuracy. The main advantage of their approach is the reduction of features by training in the latent space and simple implementation by directly handling the point cloud. Although geometric deep learning models have gained popularity in recent years, the scalability of the point cloud is necessary, where the point cloud data are much more in magnitude than the inputs currently considered in the literature. Rios et al. [36] analyzed the model and the input parameters for understanding the scalability of high-dimensional inputs in real-world applications.

2.3. K-Means Clustering

K-means clustering [37] is a simple unsupervised algorithm for learning the features without much parameter tuning and is easy to implement. An autoencoder itself is sufficient to extract features from the LiDAR point cloud. However, clustering algorithms refine the clusters, such as the deep embedded clustering (DEC) method in [38]. DEC learns the representations for clustering from the lower-dimensional data space. Coates et al. [39] conducted extensive experiments with different unsupervised learning techniques and showed that K-means outperformed other methods to solve object detection problems.

2.4. Back Propagation Feed Forward Neural Networks

Back propagation neural network (BPN) is a feedforward supervised neural network. The computation learning process happens in the forward passing of input data in the hidden layers. It uses backward passing to minimize the error between the expected and actual output of the network, reducing the bias and increasing the prediction accuracy. Back propagation networks have been applied in many research studies to solve multiclass classification problems for the past few years. Qaiwmchi et al. [40] have used the concept of metaheuristic optimization and integrated it with back propagation (BP) to determine the intrusion in the network. Melnik, O. [41] has analyzed feedforward neural networks as classification algorithms and their capabilities in segregating the input data into different decision regions. Huang et al. [42] have analyzed multidimensional data with arbitrary shapes and pointed out that the single hidden layer feedforward network with the sigmoidal activation function performed better than the multilayer network in segregating the input sample space into separate decision regions.

2.5. Model Evaluation

The most commonly used object detection and classification metrics, such as average precision, hamming loss, and coverage, are listed in [43]. Compared to these evaluation benchmarks, the real-time deployment model may not achieve the performance accuracy as the one achieved with the test dataset. Metrics such as precision, recall, F1-score, micro average, and macro average are popular for evaluating unbalanced multiclass and multilabel data, as discussed in [44].

The study’s idea is similar to the weakly supervised learning with fewer labels in [10] and group-wise learning in [16] but developed a deeper architecture for clustering 3D point cloud data in the latent space, similar to the one given in [38] for images. The self-labels created by clustering algorithms are converted into relational labels, similar to the unary and binary relations, as mentioned in [45].

3. System Design

An effective deep learning method, multilayer auto encoders, is proposed in the present research work to label the point cloud data. The neural network pipeline consists of five blocks—input, compressed representation, clustering, decoded output, and automated labeling, as shown in Figure 1.

The methodology uses unsupervised learning, consisting of autoencoders combined with K-means clustering to extract features from the point cloud data for relational class labeling and supervised learning for the segmentation and labeling of the new dataset. The input to the neural network is the point cloud data from LiDAR. Only the highly correlated feature is passed to the compressed representation block. The input features selected for the labeling and segmentation are x, y, and z coordinates with reflectance and distance of the point cloud data. The working of different blocks in Figure 1 is discussed in the below sections.

3.1. Compressed Representation—Block A

Autoencoders are unsupervised networks that learn to compress the inputs [46]. Given the point cloud input $x$, the encoder $g$ parameterized with $\theta$ learned to map the input into a fixed latent vector $z$.

$$z=g_{\theta }\left(x\right)$$

(1)

Given the unlabeled input dataset $x$ = {$x$₁, $x$₂, $x$₃,………,$x$_n}, where

$x\in R^n$ and n represent the input dimension,

$z$ is the lower dimensional hidden encoder vector of size m calculated from $x$ as follows:

$$z=\sigma \left(Wx+b\right)$$

(2)

where

$W$ is the $m\times n$ weight matrix,

b is the bias of size $m$, and $\sigma$ is the linear activation function.

In this approach, $\sigma$ is the ReLU activation function, which is defined as follows:

$$\sigma \left(y\right)=\max \left(0, y\right)$$

(3)

ReLU [43] is the rectified linear activation unit that returns 0 if it receives any negative input but returns that value for any positive value y. Thus, it gives an output that has a range from zero to infinity.

We defined the encoder as having five features as input data; the hidden layer having two times the number of inputs (e.g., 10), followed by the bottleneck layer being reduced to three inputs, as shown in Figure 2. We used batch normalization and ReLU activation to ensure the model learns well. K. Inoue’s [47] experimental analysis shows the expressive power of the ReLU activation function in the neural network.

3.2. Clustering—Block B

The K-means is one of the unsupervised learning algorithms used to classify a given data set through a certain number of clusters, when $k=\left\{k_1,k_2,k_3,\dots \dots \dots , k_c\right\}$ with $c$ cluster centers.

The distance between the cluster center and the data points is calculated as follows:

$$D_i^{\left(j\right)}=\Vert z_i^{\left(j\right)}-c_j\Vert {}^2$$

(4)

By updating the centroid for each set of $j$, $l$ is the length of the data in the cluster.

$$c_j=\frac{{{\displaystyle \sum }}_{p=0}^{l-1}{z}_j^p}{l}$$

(5)

The objective function of the clustering algorithm is given below.

$$z_j=min{\displaystyle \sum }_{j=1}^k{\displaystyle \sum }_{i=1}^n\Vert z_i^{\left(j\right)}-c_j\Vert {}^2$$

(6)

The latent-vector $z_i$ is clustered to $z_k$ with $C_j$ as the cluster center using a K-means algorithm, where k represents the number of clusters. The pseudo labeling of clusters is explained in Algorithm 1.

Algorithm 1: K-Means Clustering Algorithm

3.3. Decoded Output—Block C

The decoder learns to recreate the data points from the latent space. From the clustered latent-vector $z_k$, the decoder $f$ parameterized with $\varnothing$ learned to reconstruct the point cloud input $x^{\prime }$ that is similar to vector $x$.

$$x^{\prime }=f_{\varnothing }\left(g_{\theta }\left(x\right)\right)$$

(7)

The decoding network is similar to the encoder but with different weight, bias, and activation functions as follows:

$$x^{\prime }=\sigma ’\left(W^{\prime }{z}_k+b^{\prime }\right)$$

(8)

where $x^{\prime }$ is the reconstructed vector, $z_k$ is the clustered latent vector as shown in Equation (5), $W^{\prime }$ is the weight matrix of size $m\times n$, $b^{\prime }$ is the bias of size $m$, and $\sigma ’$ is the ReLU activation function.

The loss function used to train the AE network through the standard BPN procedure can then be written in terms of these network functions, as given below.

$$Loss\left(\varnothing ,\theta \right)=\frac{1}{n}{\displaystyle \sum }_{i=1}^n{(x_{i }-f_{\varnothing }\left(g_{\theta }\left(x_i\right)\right))}^2$$

(9)

The decoder is similar to the encoder structure, where the input layer is taken over by the latent attributes, as shown in Figure 3. The model is fitted using the efficient Adam version of the stochastic gradient descent [48] to minimize the mean squared error, given that reconstruction is a classification problem.

3.4. Automated Labeling—Block D

The clustered dataset $z_k$ in the latent space is mapped back to the input dataset $x_k$.

$$x_k=\left\{\left(x_j,L_j\right)\right\}$$

(10)

where k represents the number of clusters and j ∈ [1, k].

The dataset $x_k$ is placed in the labeled dataset sample space $S_l$ as follows:

$$S_l=\left\{x_{k1}, x_{k2},x_{k3},\dots \dots \dots ,x_{kn}\right\}$$

(11)

$S_l$ the dataset is divided into a train-test ratio of 70:30. The 70% dataset $S_{train}$ is used to train the BPN network, and 30% dataset $S_{ltest}$ is used to test the model. One must let $d$ be a single sample in $S_{train}$. After preprocessing, the data sample $d$ is represented as follows:

$$d=\left\{\left(x_j,L_j,R_j\right)\right\}$$

(12)

where

$R_j$ represents the region description of $L_j$ based on the ground truth label given in the original A2D2 dataset;

$j\in \left[1,k\right],$ where $k$ represents the number of clusters.

The trained model is tested using the $S_{ltest}$ dataset. The region-based labels generated by the model are evaluated and placed in the sample space $S_r$ as follows:

$$S_r=\{d_1,d_2,d_3,\dots \dots \dots ,d_m\}$$

(13)

where m is the number of labeled point cloud datasets.

The two-layer BPN shown in Figure 4a has five nodes as input, ten nodes in the hidden layer, and is connected to the output layer. The three-layer BPN, as shown in Figure 4b, has five nodes as input, ten nodes in its first hidden layer, one node in its second hidden layer, and is connected to the output layer. The number of hidden nodes ranged from the number of input features. The sigmoid transfer function is used as an activation function.

3.5. Label Evaluation

In traditional supervised learning, the model performance is evaluated using various metrics, such as precision, recall, F1-score, receiver operating characteristic, and other traditional indicators. However, the performance evaluation in the multilabel learning model is much more complex because each label can be associated with multiple classes simultaneously. The dataset label is not only time-consuming; even if we collect it, it also does not cover all the scenarios. The evaluation of the model using an unlabeled test data set is critical. Some of the benchmark performance evaluation metrics are given in [49].

Microaveraging is one of the evaluation methods to measure the network’s performance. In the microaveraging method, all the true positives (TPs), false positives (FPs), and false negatives (FNs) of each class in the scenes are summed up, and then the average is taken as given in the below equations.

$$\mathrm{Microaveraging} \mathrm{precision} P=\frac{{{\displaystyle \sum }}_{k_{i\in class}}TPs\left(k_i\right)}{{{\displaystyle \sum }}_{k_{i\in class}}TPs\left(k_i\right)+FPs\left(k_i\right)}$$

(14)

$$\mathrm{Microaveraging} \mathrm{recall} R=\frac{{{\displaystyle \sum }}_{k_{i\in class}}TPs\left(k_i\right)}{{{\displaystyle \sum }}_{k_{i\in class}}TPs\left(k_i\right)+FNs\left(k_i\right)}$$

(15)

$$\mathrm{Microaveraging} \mathrm{F}1-\mathrm{score} F1=\frac{2\times P\times R}{P+R}$$

(16)

$$\mathrm{Accuracy}=\frac{TPs+TNs}{TPs+TNs+FNs+FPs}$$

(17)

4. Results

The configuration files and the point cloud data are read using the python NumPy library, and exploratory analysis is performed using the python cloud library [50]. The proposed methodology is demonstrated using scenes recorded from different places.

4.1. Dataset

The A2D2 dataset [4,51] includes LiDAR, image, and bus data in the streaming sequences, recorded on highways, country roads, and cities in the south of Germany. The A2D2 dataset contains data collected from five LiDAR sensors, including ‘front left’, ‘front center’, ‘front right’, ‘rear right’, and ‘rear left’, six cameras, including ‘front left’, ‘front center’, ‘front right’, ‘side right’, ‘rear center’ and ‘side left’, and bus signals. The whole data are arranged in frames in the order of the time sequences and also define the view with the frame of reference to the ego vehicle.

Each sensor in ‘lidars’, ‘cameras’, and ‘vehicles’ has an associated ‘view’. A view is a sensor coordinate system defined by an origin, an x-axis, and a y-axis. These are specified in Cartesian coordinates (in m) relative to an external coordinate system. The external coordinate system is the car’s frame of reference. The ego vehicle contains a ‘view’ object describing the sensor’s pose for the car’s frame of reference. It also contains an ‘ego-dimension’ object, which specifies the vehicle’s extension in the car’s frame of reference. The configuration files are stored in a ‘JSON’ file and LiDAR data in the ‘npz’ format.

Table 1. Point cloud features.

Parameters	Description
X	X location in the world coordinate system
Y	Y location in the world coordinate system
Z	Z (elevation) location in the world coordinate system
Azimuth	The angle between the reference plane and the point
Distance	Distance of the point from the car’s frame of reference
Row	Pixel height location of the point in RGB camera image
Col	Pixel width location of the point in RGB camera image
Depth	Depth of the corresponding camera image
Reflectance	Reflection of the object
Timestamp	The time is taken by the reflected light

describes the LiDAR parameters recorded by the sensors.

The dataset contains semantic, instance segmentation labels, and 3D bounding boxes for 38 categories. A total of 41,277 camera images are semantically labeled, with each pixel representing a semantic category, of which 12,497 are annotated with 3D bounding boxes within the front-center camera field of view. In addition, 3D bounding boxes are estimated by human annotators, including various vehicles, pedestrians, and other relevant objects.

4.2. Model Feature Selection

The silhouette value is used to measure the clustering quality [52]. The silhouette score reaches its global maximum at the optimal value of K = 10. We used a correlation matrix to investigate the dependencies between the multiple variables. Based on the correlation matrix plot shown in Figure 5, the features’ distance and depth are highly correlated with the coefficient of 1.0, and reflectance and length are medially correlated with 0.49. Hence, we selected the five features X, Y, Z, reflectance, and distance as the input to the autoencoders. The point cloud data are filtered using the height and reflectance as proposed in the literature [18]. LiDAR point data are filtered with a Z value greater than or equal to −1.0 m (Z ≥ −1.0). The unfiltered points are clustered as road points from the procedure. This filtering procedure also downsamples the data points into a much lower dimension.

4.3. Autoencoder Training Process

The AE is trained for 50 epochs with the Adam optimizer [21] and implemented binary cross-entropy as a loss function in the network. The parameter batch size was set to 128 during training and 1 during prediction. The training process was computed on NVIDIA GTX 1080ti GPU with l6Gb of RAM. The necessary python modules, such as tensor flow [51], Keras [53], Pandas, OpenCV, and cv2, are imported. The point cloud dataset is converted to Pandas data frame for the analysis and normalized all values between 0 and 1. The dataset is split into 65% for training and 35% for testing. After 30 epochs, the AE seems to reach a stable train and test loss value of about 0.02, as shown in Figure 6.

4.4. BPN Model Training

The proposed BPN model is implemented using MATLAB R2020b. The simulations are performed on an Intel (R) Core (TM) i7-8700K CPU at a 3.70 GHz clock speed with 16 GB RAM. The model learning cycle ranged from 200 to 500 epochs. The learning rate is adaptable with the gradient boosting method for the mean square error loss function, and the momentum(mu) is set at 0.9.

Figure 7a,b show the performance of the BPN two-layer and three-layer networks. The two-layer and three-layer networks became stable at 47 epochs with the best loss values of 0.58 and 0.846, respectively.

4.5. Hyperparameter Tuning

Our 3D labeling and segmentation neural network architecture consists of block A—encoders, block B—K-means, block C—decoder, and block D—BPN.

Table 2. A concise list of selected hyperparameters for modules in neural network architecture.

Blocks in Figure 1	Parameters	Values
A	Number of Samples(N)	1024
A	Input feature dimension	5
A	Latent feature dimension (Z)	3
C	Output feature dimension	5
A, C	Number of epochs	50
A, C	Batch size	128
A, C	Learning rate	0.01
B	Number of centroids (K)	10
A	Number of neurons in the hidden layer of encoder	10
C	Number of neurons in the hidden layer of decoder	10
A	Activation function	ReLU
C	Activation function	Sigmoid
A, C	Optimizer	Adam
A, C	Loss	Binary crossentropy
D	Number of neurons in the hidden layer of BPN-two layer	10
D	Number of neurons in the first hidden layer of BPN-three layer	10
D	Number of neurons in the second hidden layer of BPN-three layer	1
D	Number of epochs	200
D	Optimizer	Gradient descent
D	Loss	Mean square error (MSE)
D	Activation function	Sigmoid
D	Learning rate	0.001
D	Momentum	0.9
D	Weight decay	0.0005
D	Learning rate decay	1 × 10−7

describes the details of the parameters in the experiment. In the first stage, we randomly sample N = 15K points from the entire point cloud of A2D2 autonomous driving scenes. After filtering the non-ground points, the subsampled point N = 1024 is passed to the training. We train the autoencoder, block A and block C end to end with the Adam optimizer, with a batch size of 128 and an initial learning rate of 0.01. Its training length is 400 epochs, but the optimal epoch value is 50 based on low training loss. The value is 0.0022, as shown in Figure 6.

We measured the clustering quality by varying the K value for the subsequent clustering of samples in the latent space with K-means in block B and using the silhouette value to find the similarity between the points in the individual cluster. We found that the silhouette score reached its global maximum at the optimal value of K = 10. We stopped the K-means when the centroids of newly formed clusters did not change. K-means is used only in the training phase for labeling, and the time complexity is less than O (KN). We train the BPN, block D end to end with the gradient descent optimizer with a learning rate set to 0.001. Its training length is 200 epochs, but the optimal epoch value is achieved at 47, since the model achieves a low mean square error (MSE) value of 0.6, as shown in Figure 7.

5. Discussion

In the present experiment, we trained autoencoders to cluster objects in the road scene using the data from the front center LiDAR fit in the autonomous vehicle. Moreover, the computer vision might have an ample supply of labeled images. For LiDAR point cloud data, obtaining labels for some fields is highly costly and laborious. Thus, unsupervised clustering of data is indeed a necessity in 3D space. The latent vector’s encoded output is sent to the K-means (k = 10) module for clustering. The clustered data are labeled by filtering the non-ground data with a Z value (Z ≥ −1.0), indicating the objects/obstacles above the road surface.

5.1. Unsupervised Feature Learning of Encoder and Decoder

In the unsupervised setting, the number of clusters within the data is not always obvious, especially when some classes may split into multiple clusters or multiple classes sent to the same cluster. Hence, decoding the image after K-means clustering in the latent space is comparatively more beneficial for labeling or identifying the data patterns. The visualization of neighborhoods of different classes on the latent space after clustering is shown in Figure 8a.

The decoded output $x^{\prime }$ given in Equation (8) is shown in Figure 8b. The model achieves a good fit in reconstructing the input, as shown in Figure 8b. As we know from the literature, labeled data for different geographical conditions are extremely rare for road scenes. Multiple output frames can be generated from the latent space without missing essential input features. Although the decoded input looks different from the ground truth image, the AE does learn important non-ground objects, dividers (1), cars (2–4), and trucks (5), as marked in Figure 8b.

Figure 8a shows the encoded and clustered output of block C, and Figure 8b is the decoded output of block C, synthesized point cloud data. This synthesized data from block C can be included in the training data for the model’s generalizability to detect new scenes. The current proposal is limited to real-time data.

5.2. Pseudo Labeling

Figure 9a,b shows the ground truth segmentation of camera images. Figure 9c,d show the BEV’s corresponding LiDAR point cloud data segmentation. The bounding boxes show the vehicles in the road scene matching the color codes given in the segmented camera images in Figure 9b. The pseudo-labeling of the non-ground data using AE and K-means is shown in Figure 9f. The pseudo labels are shown in different colors and converted to relational labels using ground truth segmentation.

Table 3. Relational labels.

Labels	Category
1	Car
2	Vegetation
3	The car is adjacent to the divider
4	The car is adjacent to the vegetation
5	Divider
6	Truck

shows the six categories of relational labels. The pseudo labels in Figure 9e, f is marked with different labels, as given in Table 3.

The 3D point cloud data segmentation with the class labels’ description shows the unified annotations of multiple classes coexisting in the same scene and the deep learning techniques proposed in the present study evolve with increasing training data.

5.3. Generating Automated Labeling and Segmentation Using BPN Network

The learning algorithm for one-hidden-layer feedforward neural networks can learn any set of observations in one short iteration, instead of many learning epochs with acceptable learning and testing accuracy, as shown in Figure 10. The network’s learning shows that any continuous bounded activation functions can form disjoint decision regions in multidimensional cases, as discussed in the literature [43]. The output of the two-layer BPN shown in Figure 10a performed better than the three-layer BPN shown in Figure 10b as it was able to label the car, vegetation, divider, and the car next to vegetation. In the case of three-layer BPN, the network mislabeled the car and divider as vegetation.

5.4. Evaluation of Segmentation (Supervised Learning)

The evaluation of automated labeling and semantic segmentation here uses the method similar to [54]. In multilabel classification, a misclassification of FPs is better than the no prediction of FNs. The false-positive counts help to optimize algorithms for the misclassified classes. The precision, recall, and F1 score are calculated based on the Equations (14)–(16), respectively, which are listed in

Table 4. Micro averaging for point cloud data segmentation.

Type of Network	P	R	F1	Accuracy
BPN two-layer	75%	75%	75%	75%
BPN three-layer	44.4%	57.1%	49.95%	50.75%

. The model accuracy is calculated based on the Equation (17). The BPN two-layer model performed the best with an accuracy of 75%.

To gain deeper insight into our model, we provided several typical cases where our approach fails on the A2D2 dataset. In Figure 10, we illustrate the labeling for different semantic categories. As observed in the output, the failures are mainly due to (i) incorrect recognition as a result of some similar pattern between the object classes (e.g., vegetation and divider) or (ii) misclassified small or occluded objects (e.g., truck or car). We compared our result with other state-of-art methods in a similar field of study. In group-wise learning [16], the segmentation accuracy of images is 68.7%; for point cloud, weakly supervised segmentation with fewer labels [10], the segmentation accuracy of a real-time point cloud dataset S3DIS (Area 5) is 48.0%. In the same study area as ours, the accuracy of different datasets shows that more labeling techniques are required to improve the segmentation task for group-wise learning with unsupervised learning methods.

6. Conclusions

In the present research, 3D point cloud data labeling is considered semi-supervised learning, as it is streamlined using both unsupervised and supervised learning models. The geometric properties of data lie in the lower dimension rather than the actual input space, and each data point has a set of neighbors connected by Euclidean distance. In the proposed architecture, the unsupervised learning model labels the poorly labeled data into the relational label. Dimension reduction is much better than the down-sampling techniques, since it preserves spatial information without losing the data patterns. The experiment reduces the original point cloud data dimensionality to 30% using autoencoders. The decoder generates new images, and K-means creates the subset named pseudo labels of the n-dimensional Euclidean space. The synthesized image can help improve the generalizability property of the model for segmenting new scenes.

Each pseudo label is compared with the ground truth and converted into relational labels. These relational labels are also called class labels. The backpropagation feedforward neural networks are trained using these class labels to label the unlabeled data. The labels are evaluated by finding the true positive, false positive, and false negative. To improve accuracy, the supervised model uses multiple learning iterations with more labeled data and optimization techniques. The two-layer BPN model performed better than the three-layer BPN model.

The synthesized data can also be included in training the BPN models for future investigations. The study offers an opportunity to improve the methodology or application further to merge multiple frame receivers from other sensors by observing similarities and dissimilarities, and labeling the relationship between different frames. While the present work focuses on clustering and labeling models from BEV, it is possible to extend the models to incorporate data across multiple views from LiDAR. By including point cloud augmentation algorithms, rotation, and homogenous transformation matrixes, we can change the viewpoint to a global coordinate system in the model hyper parameters, and integrate AE with rotational and angle invariant transformer networks, such as spatial transformer and T-net models.

The labeled dataset can be used in supervised learning for scene segmentation. One of the downsides of the methodology is that we have to create our initial clusters based on the data encoded by the untrained network. The clusters may not necessarily be as accurate but can still produce semantically meaningful groupings. Another downside of the method is that it requires the number of clusters as a hyper parameter. In the future, the approach can be extended to incorporate graphical neural networks to learn the relationship between the neighboring points using node classification and segmentation.