Online Learning for Dynamic Impending Collision Prediction using FMCW Radar

2.1 Problem Statement

We consider a mobile node which has an attached FMCW radar, an inertial sensor to measure node’s acceleration, and a processor for computing.

The mobile node is moving in an environment with several obstacles. Let us consider the scenario where the radar-enabled node is moving towards an obstacle. Let \(d(t)\) be the range and \(v(t)\) is the relative velocity between the mobile node and the obstacle at time t. During one measurement time period between \(t_1\) and \(t_2\gt t_1\), duration \(T = t_{2} - t_{1}\), the node collects samples from the radar and inertial sensor. At time \(t_2\) it converts them to a range-Doppler matrix, \({X}_{t_{2}}\).

The goal of our system is to raise an alarm at \(t_{2}\) if at any time \(t\in [t_{2},t_{2}+\delta t]\), that \(d(t) \lt \epsilon\), where \(\delta t \gt 0\) is the time duration into the immediate future for which we must detect a future collision. Also, \(\epsilon \gt 0\) is a collision proximity range threshold. In other words, if the measurement indicates that the node will collide with the object within the next \(\delta t\) period of time, the alarm should be raised. In order to achieve this, we create a system that can learn a function f that maps from the range-Doppler matrix to a binary decision about whether there will be an impending collision.

2.2 Learning Framework

For the collision prediction problem, the inputs we use are RDM matrices, named X, that get mapped via the function f to \(\hat{y} \in \lbrace 0, 1\rbrace\), where class 0 is encoded as 0 for representing ‘no impending collision’ and class 1 is encoded as 1 representing ‘impending collision’. This makes collision prediction a binary classification problem. Supervised learning-based approaches require a training set of ‘measurement-label’ pairs \([(X_1, y_1),\ldots (X_n, y_n)]\) where n is the total number of pairs in one training set, \(X_j\) is matrix j and \(y_j\) is its collision label, such that \(j \in \lbrace 1,\ldots ,n\rbrace\).

Loss Function

In order to learn the optimal mapping f between RDM and \(\hat{y}\), a classifier for two classes needs to minimize a loss function L for the entire training set of size n, given by (1) \(\begin{equation} L = - \frac{1}{n}\sum ^n_{j=1}[{y_j}\log (p_j) + (1-y_j)\log (1-p_j)], \end{equation}\) where \(p_j\) is the probability of the \(j^{th}\) data point belonging to a class as predicted by the mapping. Taking an average over the entire dataset of size n, we get cross entropy loss L, a standard loss function for classification problems.

CNN-based Classifiers

CNNs have become widely popular for image-based machine learning tasks. Compared to the standard MLP architectures, a CNN architecture uses far fewer parameters and can have much deeper architectures which can equip us to solve more complex problems. There are several CNN architectures available with varying degrees of computational complexity and performance.

For our learning-based solution, we test two CNN architectures: (a) ResNet architecture, since it has shown the top-5 error rate on ImageNet dataset [27] and (b) MobileNet-V3 architecture, since it has been optimized for low-resource devices and low latency applications [28].

2.3 Radar Signal Processing

The FMCW radar transmits sequences of a linear frequency modulated (LFM) signal, also called a ‘chirp’ signal having N samples per chirp, which increases its frequency linearly with time with a bandwidth of B Hz and chirp duration (also called pulse repetition interval) of \(T_c\). The slope s of the chirp represents the rate of change of frequency, thus \(s = B/T_c\). Once the radar receives the reflected signals that bounce back from the target, they are mixed with the transmitted signals to obtain a ‘beat signal’. The beat signal is characterized by the beat frequency, \(f_b\), which is equal to \(2ds/c\), where d is the range of the object from the radar and c is the speed of light. This frequency is used to infer the range of the target from the radar sensor. A fast Fourier transform (FFT), called the ‘range-FFT’, is performed on the beat signal to convert it into the frequency domain, thereby obtaining the beat frequencies representing one or multiple objects at various ranges, with a range resolution of \(c/2B\) and up to the maximum range of \(f_{s}c/2s\), where \(f_s\) is the sampling rate, such that \(f_s =N/T_c\) [51]. A second FFT, called the ‘velocity-FFT’, is then performed across a certain M number of chirps that make one RDM, to estimate the relative radial velocity. It is left to the discretion of the designer to define how many of the chirps are going to be processed together into an RDM (due to processor limitations and/or resolution requirements) [20]. Our proposed method implements this 2-D FFT stage and generate a \(N\times M\) matrix, which is referred to as the radar-Doppler matrix (RDM), that indicates the amplitude of scattering from each possible relative range and velocity across the possible range of measured range and velocities.

It should be noted that reflection and scattering of the radar signals by objects are a function of the intrinsic properties of the material of the objects, such as dielectric permittivity, magnetic permeability, and electrical conductivity [63]. These properties have been shown to be effective in object distinction and detection [73]. Additionally, the extrinsic characteristic such as the absolute size of the objects and the size of the objects relative to the wavelength of the radar signals play a crucial role in the amplitudes of scattering received in the radar reflections. Therefore, different types of objects produce different amplitudes of scattering within RDMs.

2.4 Label Creation

Our proposed system automatically generates (with a sub-second delay) labels for the collected radar data using inertial sensing. This labelled training data can be used to automatically retrain the model in order to improve results for the particular environment and user characteristics that are observed during operations.

As a collision between the moving node and an obstacle occurs, the moving node experiences a sudden change in velocity. This change in velocity can be clearly observed as a change in the measured acceleration, for example, as a sharp drop or negative peak in mobile node’s IMU measurements, as seen in Figure 2(a). During normal operation we obtain these ‘moments of collision’ by the finding peaks in a node’s measured acceleration data.

Fig. 2.

View Figure

A collision prediction system should alert the nodes about impending collisions before they happen such that the colliding nodes have time to act preemptively to avoid the collision. Considering an alert can be raised instantaneously by a collision prediction system without any mechanical or processing delay, a node receiving this alert, however, will take time equal to its reaction time for any responsive action. To have at least one alert sent by the system to the node before any impending collision, the immediate future \(\delta t\) required to check for impending collisions must be at least equal to the reaction time of the node. Thus, if a collision occurs at \(\delta t\) into the future, it can be avoided or ameliorated by issuing one alert at a time that is \(\delta t\) before the collision. Depending on the reaction time of the node and the number as well as the frequency of alerts needed for the node (robots or human) in any specific application (vehicular or sports-related), the threshold for \(\delta t\) can be adjusted. Based on this approach, for labelling purposes, all the measurements (the RDMs in our case), that happened within \(\delta t\) from each moment of collision as measured by IMU are labelled as ‘Impending Collision’ (shown as magenta colored square markers on trajectory as shown in Figure 2(b)). The rest of the RDMs are labelled as ‘No Impending Collision’ (shown as blue colored circle markers as shown in Figure 2(b)). The labelled images are then used to train our CNN-based model at the subsequent training time to build the next model that provides inference about impending collisions when encountering new, unseen test RDM samples.

It should be noted that in real-world scenarios, false labelling is possible, where, for example due to sensor noise or outdated thresholds, RDMs can be labelled incorrectly through the current labelling scheme. This leads to ‘label-shift’. Deep-learning models tend to be robust against infrequent and randomly mislabeled data, treating the labels as noise or outliers. The ability of a model to treat a small number of incorrect labels as noise depends on the quality and quantity of the noisy data, and the overall complexity of the problem being solved. Therefore, precautions are suggested in the choice of IMU sensors and the threshold of the deceleration for the application. Other traditional techniques can also be adapted to handle the label noise prior to training such as label smoothing, example weighting, and custom loss functions [32].

2.5 Class Imbalance Problem

During system operation, the radar is continuously sensing the environment. Even in the environments that are densely populated with obstacles, the events during which a collision is imminent is considerably lower in frequency than the frequency of events in which the node is not in danger of an imminent collision. With the labelling scheme for the generated RDMs described above, the number of RDMs labelled as ‘impending collision’ are order of magnitudes lower than the number of RDMs labelled as ‘no impending collision’, which leads to an imbalanced dataset.

Imbalanced datasets, such as the state-of-the-art Give-Me-Some-Credit dataset [33] and statlog heart disease dataset [67], are common in anomaly detection, medical diagnosis, fraud detection, and are challenging to tackle in the machine learning domain, as the standard learning algorithms may struggle to perform well on minority classes due to the bias introduced by the majority class. There are many methods to tackle this challenge: re-sampling methods which include over-sampling of the less frequent class using random, SMOTE (Synthetic Minority Over-sampling Technique), and ADASYN (Adaptive Synthetic Sampling) techniques, or under-sampling of the more frequent class using random and Tomek links [13], [26], and [66]. However, important considerations needed to be weighed before applying these. For example, in over-sampling, increasing the size of the training set without gaining any additional information might not add any benefit, and can introduce noise. On the other hand, under-sampling might lead to discarding of the relevant information. Other methods include the use of tree-based algorithms or ensemble methods like bagging and boosting, which might lead to over-fitting [31] and hence poor generalization to new data. There is no one-size-fits-all answer to which method is better for addressing imbalanced datasets, as the effectiveness of different methods or combination of different methods depends on various trade-offs and factors such as computational cost, interpretability, and evaluation metrics. For our system, we refrain from using the over-sampling techniques as they might increase the size of the dataset without adding any useful information, thereby increasing the computational load on the resources required for training the model in general. Furthermore, under-sampling methods are not explored as a possible solution for our system’s imbalanced dataset problem as under-sampling may remove the data points that otherwise would provide the highest uncertainty during inference, a required criteria for our retraining framework, such that we can make our system to be robust against data-drift.

Instead, we use the method proposed in [37], in which each class is assigned class weights that is inversely proportional to their respective frequencies, such that (2) \(\begin{equation} w_j= \frac{|C_1| + |C_0|}{2 |C_{j}| }, \end{equation}\) where \(|C_j|\) is the number of samples in class \(j \in \lbrace 0,1\rbrace\). Assigning a small weight to the cost function for the more frequent class during training results in a smaller error value, and thus, a small update to the model coefficients for the more frequent class. A large weight applied to the cost function for the less frequent class will result in a larger error calculation, and in turn, a large update to the model coefficients for the less frequent class. This way, we can shift the imbalance of the model so that it could reduce the errors of the less frequent class. Choosing this method provides highest interpretability compared to other methods, while it also maintains original distribution of instances and preserves the inherent patterns and relationships in the data.

An additional problem encountered for imbalanced datasets is that using the accuracy of the model to assess its performance becomes a less reliable metric because an acceptable accuracy is possible to obtain even if the model has excellent classification performance for more frequent class and poor performance on the other, less frequent class [38]. Thus, instead of using accuracy to pick the best model, we use the F1 score as a metric that describes the performance of the classifier on both the classes. F1 score is the harmonic mean of the positive predictive value (P, also called precision) and true positive rate (R, also called recall or probability of detection (\(P_D\))) given by (3) \(\begin{equation} F1 = \frac{2 P R}{P+R}, \end{equation}\) where P is the fraction of correctly classified positive instances (‘impending collision’ in our case) among the overall instances that are classified as positive, while R is the fraction of correctly classified positive instances among the overall instances that are truly positive. A high F1 score indicates low misclassifications in both of the classes. Therefore, during training epochs, we select the model which has the highest F1 score.

2.6 Online Learning Framework

Real-world implementation of any collision system experiences a continuous influx of new data in real-time. A static learning-based solution leads to decreased inference performance, especially if the new data is from different distributions, as a result of, for example, changing user behaviour or environmental conditions compared to the previous data on which the model was trained.

This phenomena is called data-drift, which includes changes in the distribution of the data (co-variate shift), changes in the distribution of the labels (label shift), and changes in the distribution of the relation between data and labels (concept drift) [29]. Domain adaptation, transfer learning, fine-tuning, and continual learning enable models to adapt to the changing data distributions and maintain good performance, however, these methods need a large amount of the dynamically changing data. On the other hand, meta-learning provide solutions that have advantage of working with only a few or single instances of new data [18]. However, if the data distribution of the target task significantly deviates from the data distribution of the training tasks, the performance of the meta-learning methods model may degrade and hence, are not seen as a better alternative [72, 76]. Moreover, since our system in question experiences abundant data that has undergone distribution-shift, there is value in utilizing all the relevant data in our application. In order to maintain the model’s inference performance over time on new, unseen data that has experienced data-drift, one of the techniques is to periodically retrain the model to tackle data-drift in real-world applications. Model retraining is defined as re-running the process that generated the previously selected model, however, on a new training dataset that has experienced data drift. This can be computationally expensive, thus, we explore the branch of active learning and suggest the following retraining strategy:

• Continuously collect the system’s IMU-labelled RDMs from recent history to form a retraining dataset.

• Based on the system requirement for the retraining latency, select a fraction of the collected dataset using the uncertainty-sampling technique [40].

• Periodically retrain the default or current model using this newly sampled subset of data.

• Repeat the above steps for the newest recent history of the dataset collected.

It should be noted in real-world scenarios that using the automated labelling scheme can experience noise or ‘label-shift’, which further leads to ‘co-variate shift’ [29]. Therefore, it is necessary to periodically retrain the model that is implemented in dynamic environments. We explore how to implement such a scheme for resource-constrained devices in Section 4.4.

3.1 Objects and Environment

In order to obtain a large quantity of data in which our collision prediction and warning device experiences actual collisions with obstructions, without having to subject a person to carry the device and experience the same collisions, in this paper we implement the following setup. We use a robotic motion platform as our moving node for our experiments that is comprised of an iRobot Roomba, which is controlled by attached Raspberry Pi-3 module as shown in Figure 3(b). The node moves in a laboratory environment as shown in Figure 3(a) which has other stationary obstacles. The OptiTrack motion capture system [50] tracks and records the ground truth position co-ordinates of all the objects that are tagged with reflective markers. An 3 m \(\times\) 3 m area is barricaded with PVC pipes in order to confine the moving node and maximize its number of collisions with obstacles within the observed area. Within this obstacle-rich area, randomly placed obstacles are present that are filled with a variety of materials. In order to collect a dataset with a variety of radar reflections for our collision prediction method, we choose three different kinds of materials (gravel, soil, and water), as shown in Figure 3(c), 3(d), 3(e), and 3(f), to simulate the items that are commonly encountered by a moving objects (vehicles or humans) in the real-world. Cylindrical shaped objects are 0.5 m in height and 0.3 m in diameter, whereas rectangular shaped objects are \(0.5 \times 0.2 \times 0.3\) m in width.

Fig. 3.

View Figure

3.2 Sensing Hardware

We use AncorTek’s SDR-KIT 2400AD2 [68], a low-power and compact software-defined radar for FMCW-based sensing, sitting at the top deck of the prototype. The center frequency of transmitted signal is adjustable within the 24–26 GHz frequency band. AncorTek provides drivers only for Windows, and thus, we use a Windows laptop to receive and store the complex in-phase and quadrature (I/Q) components of the received FMCW radar signal. For the radar’s settings, we choose a bandwidth of 2 GHz at a center frequency 25 GHz during all our experiments. Each chirp is of 1 ms duration with \(N = 128\) samples per chirp. With these settings, the range resolution and the maximum range that can be measured are 0.075 m and 4.8 m, respectively. Also, the maximum radial velocity that can be measured is 3 m/s. The radial velocity resolution is dependent on the number of chirps processed together at once for one RDM [20]. Lastly, the moving node also has a BNO055 IMU sensor [64] that collects the acceleration data during our experiments at the rate of 60 Hz. It is noted that this low-cost IMU sensor is prone to noise and thus we use the highly accurate OptiTrack position ground-truth coordinates for estimating the mobile node’s higher order kinematics, such as acceleration, which is taken as the second derivative of position with respect to time, and is used for the labelling purposes. All the data files are provided [60]. It should be noted that the size of the radar sensor used for our experiments is \(79 \times 56 \times 13\) mm, which is comparable to the size of a commercially available collision warning radar systems. However, in applications such as smart wearables and smart devices, the size of sensing hardware is encouraged to be miniaturized, for example, as accomplished through Google Soli’s 12 \(\times\) 12 mm radar chip [41].

3.3 Motion and Collisions

We conduct a series of experiments with one mobile node moving in a cluttered environment with eight obstacles that are stationary. The maximum velocity that can be reached by the moving node is 0.5 m/s. The moving node is made to travel in two kinds of trajectories, straight and curved. For the straight trajectory, both the wheels of the moving node are programmed to maintain one constant speed (0.5 m/s), and thus, the node moves in a straight line motion. For the curved trajectory, one wheel of the moving node rotates slower than the other, creating a curvature in the trajectory, and mimicking a curved line motion. Several collisions happen between the mobile node and the obstacles during both styles of these trajectories as given in Table 1. When a collision happens, the moving node comes to a complete stop, takes a turn by a random angle, and then starts moving again at the same speed and in the same style of programmed trajectory as before the collision. Lastly, we do not consider the PVC pipes as ‘obstacles’ since they are used only to keep the robots inside the experimental area, and thus, all the collisions and the corresponding RDMs between the moving node and the PVC pipes are not included in the final dataset.

3.4 Data Characteristics

We run 10 experiments, each one either with curved or straight trajectories and each experiment is of 10 minutes duration. The duration of measurements, and thereby the size of the dataset, affects the convergence time and generalization performance of the model; however, the amount of dataset recorded, and stored is dependent on available resources and requirements of the system. The maximum duration possible to continuously record the IQ samples through AncorTek GUI is 10 minutes. Each experiment of 10 minutes thus creates one dataset that are shown in Table 1. We process \(M=200\) chirps together to make one RDM, thus for our experiments, one RDM comprises of 200 ms duration of complex-valued radar samples. Moving the RDM window every 50 ms, one 10-minute experiment generates \(1.2\times 10^4\) RDMs. The values for M and sliding window size are adjustable as suited by the system’s requirements, such as desired velocity resolution and available computational capabilities to process and store RDMs, as is further investigated in Section 4.2 and Section 4.4. We drop all the collisions with PVC pipes and their corresponding RDMs, as described above, thus obtaining \(6\times 10^3\) RDMs per experiment on average. On average, 32 collisions are encountered during one experiment. With the labeling scheme described in Section 2.4 and taking \(\delta t = 400\) ms, we obtain on average 340 RDMs per experiment that can be labelled as ‘impending collision’, while rest of the RDMs are labelled as ‘no impending collision’. Since only 3% of the RDMs can be labelled as ‘impending collision’, this dataset is highly imbalanced. As explained in Section 2.5, we deal with this issue with the help of Equation (2) by assigning weights to each class, as given in Table 2.

3.5 Radar Sensing for Traditional CWS

Environments where collisions are likely to happen are characterized by compact spaces with numerous and a variety of objects. Radar systems operating in such environments suffer from additional reflection and scattering from objects which do not cause collisions. For example, scattering from the ground contributes to reflections received at the radar, but in many applications, the node is not going to collide with the ground. Another source of unwanted reflections is the physical object to which the radar is attached, which, for example, in the smart helmet case includes the person themselves.

Overall, we refer ‘clutter’ as the received radar reflection that are due to any objects within the environment with which the moving node can not possibly collide. Typically these reflections make the largest contribution to the reflections received by the radar and hence, the visibility of the relevant targets in the radar images with which collisions risk must be addressed, get suppressed. Therefore, traditional radar-based CWS systems need to filter out unwanted reflections due to clutter before obtaining range and velocity information.

Classical principal component analysis (PCA) has been extensively used in image and video processing applications as a statistical tool to seek the best low-dimensional approximation of the high-dimensional data. We apply an advanced version of PCA as a model-based filtering method in order to remove reflections that are due to the clutter and extract the reflections that are due to the object from the RDMs.

The data can be constituted as a superimposition of two components: (1) L, which is the low-rank matrix and (2) S, which is a matrix that can be sparse or not. This decomposition can be obtained by robust principal component analysis (RPCA) solved via principal component pursuit (PCP) [5]. Due to the correlation between RDMs, the background and clutter are modeled by a low-rank subspace that can gradually change over time, while the objects that are in relative motion with respect to the radar constitute the correlated sparse outliers represented in the sparse matrix.

We demonstrate with the help of Figure 4, the effect of applying RPCA-PCP to RDM images for the three RDMs for various scenarios: (a) radar facing no object in its field of view, (b) radar facing one stationary object in its field of view, and (c) radar facing two moving objects in its field of view. In the left column of Figure 4, the bright vertical reflection at zero velocity is due to clutter. This clutter reflection is removed with the help of RPCA-PCP reconstruction of the signal, thereafter showing only the target in the RDM, as shown in the right column of Figure 4. The post-PCA, filtered RDMs are then processed by the range-FFT and velocity-FFT as explained in Section 2.3 to obtain range and radial velocity of the target. It should be mentioned that the amplitude that represent the target gets reduced after the PCA-based clutter removal step. It should also be noted that this clutter removal step is a requirement only for the baseline method. Our proposed CNN-based solution does not require this additional step of clutter removal since it uses the unfiltered RDMs as input for the training as well as for the testing stages. Lastly, since all the obstacles get represented by the sparse S matrix, the PCA method successfully filters out the clutter subspace and retains the subspace that is due to obstacle(s), such that reconstructed RDMs can show all and any number of objects within RDMs, as is shown in Figure 4(c).

Fig. 4.

View Figure

4.1 Training Performance of Proposed Method

Our goal is to train a classifier which takes RDMs along with their corresponding labels as input for training and predict the label for unseen RDMs. The cross entropy function specified in Section 2.2 is optimized in this process of training the CNN. For training the ResNet-18 CNN as the classifier, Adam [36] is used as the optimizer to update the model’s parameters. We also train the MobileNetV3-Small model on the same training dataset and use the RMSProp optimizer [65]. We use D as our training dataset that has all the RDMs from four experiments in total: Exp. S1, Exp. S2, Exp. C1, and Exp. C2. The learning rate for the both the models is set to be \(5 \times 10^{-4}\) and both of the models converge in 30 epochs.

In order to demonstrate that the features space of the RDM dataset is successfully learned by ResNet-18, to be able to distinguish RDMs into two classes based on the dynamically changing distribution of features, we create a visualization of the features space as follows: Each RDM from training dataset D is fed to the ‘untrained’ ResNet-18, creating a 512-dimensional features space of each RDM in the fully connected layer before the final classification. We then use the t-distributed stochastic neighbor embedding (t-SNE), a popular dimensionality reduction technique, commonly used for data visualization for visualizing high-dimensional features in lower-dimensional space [69]. The parameters for creating t-SNE embeddings using the sklearn library are kept as learning_rate=auto and init=random. The obtained 2-dimensional t-SNE embedding of the 512-dimensional features space from an untrained ResNet-18 model are shown in Figure 5(a). We see that the untrained ResNet-18 model cannot separate the features space of the RDMs into the two desired classes. Now similarly for the ‘trained’ model, we obtain the 2-dimensional t-SNE embedding of the 512-dimensional features space, as shown in Figure 5(a). We see that the trained ResNet-18 model is now able to linearly separate and cluster the RDMs in the combined dataset into collision and non-collision classes, indicating a successful learning of the features space.

Fig. 5.

View Figure

4.2 Inference Performance of Proposed Method

For the inference performance of the trained model on any unseen data, we first provide a visual intuition into the collision prediction system with the help of Figure 6.

Fig. 6.

View Figure

In the two sub-figures, the surroundings of two different obstacles named ‘Object-2’ in Figure 6(a) and ‘Object-3’ in Figure 6(b) are shown, along with the straight and curved trajectories traversed by the moving node in the unseen data from Exp. S3 and Exp. C3, respectively. The blue arrow shows the direction of motion which leads to collision incidents between the stationary obstacles and the moving node. The location of the moving node at the time one RDM gets measured is shown by one dot (\(\boldsymbol {\cdot }\)). During the inference stage, an RDM is fed to the trained model and a prediction about the ‘impending collision’ is made. The steps for which ‘no impending collision’ is predicted are as represented by blue (\(\boldsymbol {\cdot }\)) dots, and the steps for which ‘impending collision’ is predicted are represented by orange (\(\times\)) cross. The goal of Figure 6(b) is to provide a visual representation of the inference performance made by our trained model, which includes correct and incorrect classifications for both of the classes.

We combine the datasets from Exp. S3 to S5 and Exp. C3 to C5 to use as the test dataset T, that has the RDMs that are not seen by the trained models. For this test dataset T, predictions made by the two trained models (ResNet-18 and MobileNetV3) are collected and compared with the ground truth for collisions according to the IMU’s deceleration-based labels. We also investigate the performance of the traditional supervised learning methods such logistic regression (LR), k-nearest neighbors (k-NN), and support vector machine (SVM) using the same datasets D and T for training and testing, respectively. Figure 7 shows the receiver operating characteristic (ROC) plot for the all the five investigated learning-based classification models as well as the baseline model. The y-axis is probability of detection \(P_D\) (or recall R as described in Section 2.5) which measures the fraction of RDMs that are inferred as ‘collision’ over all the RDMs that were truly labelled as ‘collision’ during the \(\delta t\) window from moments of collisions, as was done in the labelling scheme of Section 2.4. The x-axis represents the probability of false alarm, \(P_{FA}\), which measures the fraction of RDMs inferred as ‘collision’ over all the RDMs that were truly labelled as ‘non-collision’. The ROC plot shows that ResNet-18 has the highest area under the curve (AUC) with 0.98, outperforming all the other classification methods and thereby, being our investigation’s suggested classifier for the collision prediction problem. The reported F1-Score is 0.91. The MobileNetV3 gives the second best AUC (0.88). The traditional supervised learning methods have AUC of 0.78 for SVM, 0.78 for kNN, and 0.74 for LR, while the baseline method of combining PCA-based filtering with the TTC model has an AUC of 0.6.

Fig. 7.

View Figure

We further investigate the nature of predictions made by ResNet-18 on the test dataset T, by exploring the prediction accuracy as a function of the time to collision. As seen in Figure 8, all the false alarms raised by the ResNet-18 classifier are more frequent near the time of the actual collision. This indicates that most of the false alarms (false positives) are close to the time of the threshold of \(\delta t\). That is to say that the false alarms are less likely to occur when the moving node is further away from objects on the trajectory. Additionally, it is reported that the probability of detection is the nearly consistent when the time of collision is less than the threshold of \(\delta t\), indicating that the classifier is able to detect the impending collisions at the specific threshold of \(\delta t\) as designed for our experiments.

Fig. 8.

View Figure

We also investigate the effect of the value of M, that is the number of chirps used to make each RDM on the predictions accuracy of ResNet-18. For this, the radar and IMU measurements from experiments in dataset D and T are processed for varying M values. The frequency of RDM generation is stays the same, that is, the RDM window moves by 50 ms at a time, thereby keeping the dataset size the same for comparison between models processing different RDM window sizes. All the other parameters of measurements s, B, \(T_c\), and N, also stay the same, thus, the direct effect of varying the M is in getting a varying velocity resolution information in RDMs. A larger value of M results in more chirps getting combined to make a RDM, which increases to the processing and computational cost. For predicting a collision, this implies having an RDM that has chirps of a longer time period before the collision. We report that the AUC score increases with the value of M, as seen in Figure 9, thereby indicating that having more chirps, which provides more past data and finer velocity information in RDMs, leads to higher learning and decision making performance at the cost of additional computational cost. However, there is an optimal value for M, beyond which the performance degrades or plateaus as the longer the duration of RDMs, the older the chirps are in the RDMs. It can be inferred that the accuracy of predictions about future decreases as the age of the data increases because the outdated information can become irrelevant to the immediate future.

Fig. 9.

View Figure

Lastly, we also report the effect of the materials of the obstacles on ResNet-18 model’s prediction capabilities. The radar reflections vary with the material of the obstacle but our collision prediction model performed with a \(P_{D}\) of 0.95 or higher, regardless of the material composition of the obstacle as shown in Figure 10. For the test datset T, we conduct an one-way ANOVA test. With the null hypothesis being that the materials of the object do not affect the probability of detection of our model and the significance level or Type-I error of 0.01, we report a p-value of 0.0254. Thereby, we accept the null hypothesis, indicating that our model’s performance does not vary with the material of the encountered obstacles.

Fig. 10.

View Figure

4.3 Online Learning Performance

Following the retraining strategy in Section 2.6, the ResNet-18 classifier is investigated in its retraining performance on the previously unseen data. First, a default model is obtained by training a ResNet-18 classifier on one of the straight-trajectory datasets Exp. S1. In order to emulate data drift, a new, unseen dataset from one of the curved-trajectory experiments Exp. C1 is used to obtain the performance of the default model. As presented in Figure 11, the default model’s classification performance on the new, unseen dataset gives an AUC value of 0.69. Next, this curved trajectory dataset is considered as the recent history and \(80:20\) train-test split of this recent history is then used as our training set to retrain the default model. The retrained model generates an AUC of 0.95 on the test set if the full training set is used. It should be noted that large datasets require a long training time. Therefore, we apply the uncertainty-sampling technique for selecting uncertainty-based samples from the training set for retraining. We present results for various sample sizes selected for retraining, for example, using the top 10% of samples from the full training set that showed highest uncertainty to form as a subset of training set for retraining generates an AUC of 0.83. Similarly, using the top 20% and 40% of samples with highest uncertainty generates AUC of 0.884 and 0.885, respectively. We report that the AUCs obtained by retraining on the data subsets of various sizes of the recent history are higher than the AUC from the default model that was not retrained. Furthermore, using uncertainty-sampling enables low latency and data requirements for retraining and provides a better performance than a static, default model.

Fig. 11.

View Figure

4.4 System Design and Limitations

This section discusses how to apply our system for real-time operation in a resource constrained wearable device.

There are multiple steps in the processing pipeline of the proposed system. First, a base model is trained using prior data collected beforehand. This training is presumably performed in the cloud using a cloud-offloading approach, similar to [54]. Thereafter,

We have used an Intel-Xeon E52666 (CPU) for the processing of radar signals, but this step can be performed on any embedded microprocessor in real-time. For example, in common smart wearables, a 1.6 GHz Quad-A7 Snapdragon is the system-on-chip (SoC), which is reported to generate every second a total of 1200 RDMs that are of similar configuration as our experiments. This corresponds to a latency of 0.8 milliseconds per RDM [41].

To run the deep-learning pipeline on resource-constrained smart devices, several libraries provide run-time optimizations to accelerate the inference performance of models [53]. The overall system design for smart wearables can benefit from exploring the options in terms of available SoCs and their performance [75]. In terms of computational cost, advanced architectures of CNNs, such as [15], can provide alternatives, however, a comparison to such recent and advanced architectures in terms of accuracy is beyond the scope of this study.

In terms of communication, smart wearables are known to support NFC, WiFi 802.11ac, and Bluetooth LE and can communicate with any local device with powerful GPUs and CPUs. The number of wearables that can directly connect with the cloud is increasing as the demand grows [52]. Apart from the communication delay of computational off-loading, the overall processing time for one inference is primarily composed of on-device model’s inference latency, which is dependent on the model of choice, deep learning libraries, and the hardware.

As an example of the relative inference latencies, we compare the two CNN architectures’ inference latencies for one RDM on Intel-Xeon E52666 in Table 3. It can be seen that MobileNetV3 achieves a 16 times reduction in the model latency compared to the widely popular ResNet-50 and 6 times reduction compared to our work’s proposed model of ResNet-18. The CNN architectures also differ in their number of trainable parameters. MobileNet architecture has a 10 times lower number of parameters to train in comparison to the ResNet architecture. Therefore, we suggest MobileNetV3 for solutions where swifter predictions are required to be made while having better performance than traditional supervised learning methods. Lastly, for implementing our design on smart devices, MobileNet architecture has shown similar inference performance on SoCs such as Snapdragon 430 to 888, thereby making our suggested architecture a suitable option for smart devices and mobile applications [75].

The above suggestions are in accordance with the current challenges faced by smart devices, in that the majority of them are resource-constrained in terms of computational and communication capabilities. The system design for a smart wearable such as smart helmet, thus requires innovative schemes to implement our method’s pipeline and suggestions to capitalize on available resources.

5.1 Smart Helmets and Collision Prediction

Two of the most relevant fields that deploy preventative measures similar to our problem statement of predicting and avoiding collisions are of smart helmets and vehicular CWS. Smart helmets have gained substantial interest from the research community in the past decade. The goal of smart helmets can be diverse, most commonly used for collision prevention, by either deactivating the paired vehicle if helmets are not in the vicinity [56] or by deciding that a collision has occurred and call for help after the collision event [46]. They do not actively predict impending collisions before they happen. Therefore, there is scope for incorporating the smart helmets with advanced sensors and algorithms to instead predict the collisions before they happen, which is crucial in altering the wearer of these helmets to prepare themselves upon receiving an alert according to the prediction. Efforts for a camera-based collision prediction algorithm is presented in [8], where future positions are ‘extruded’ and the decision about a future collision is made. Another camera-based helmet for avoiding rear end collisions in real-time is presented in [58]. Most of the work in smart helmets utilize cameras or a combination of vibration sensors. To best of our knowledge, solely using radar-based smart helmets for predicting collisions before they happen is novel.

5.2 Vehicular CWS and Radars

Vehicular CWS, on the other hand, are well researched and are adapted as a necessity for the automotive industry. The automotive industry commonly utilizes the 24 GHz and 77 GHz frequency bands in radar systems [22]. Both frequency bands have their advantages and limitations. The choice of sensor for the vehicular CWS is most commonly a combination of camera and radar sensors. Radar performance may be degraded performance due to clutter and noise. In order to minimize the clutter reflections received and noise captured in these radar sensors, filtering techniques such as [6] are actively being researched. However, these filtering methods are not dynamic and need to be updated frequently whenever the environment in which the radars operate changes. Instead, deep learning can be utilized to work with measurements from radars that operating in cluttered and dynamic environments. Multiple published papers that are using deep learning on radar measurements have resulted in successful learning-based solutions in the field of motion and object detection, classification [35], localization and tracking [3, 42, 43, 77], and human health and activity monitoring [14, 55].

5.3 Vehicular CWS and Deep Learning

Although the progress in deep learning have found much application in radar-based sensing, the learning-based algorithms for CWS are still being developed. In [62] and [19], the authors present a non-learning method in the former and a CNN-based classifier in the latter to use a combination of camera and millimeter-wave radar and predict collisions. Instead of using sensing hardware like radars, another option is to provide input information is vehicle-to-vehicle (V2V) or vehicle-to-infrastructure (V2I) communication streams for obtaining range and velocity information for collision predictions. Based on V2I communication, in [39] a collision warning algorithm using a multi-layer perceptron (MLP) neural network called MCWA is presented and reported to have higher performance compared to non-learning methods. A more advanced architecture using VGG-CNN, called RCPM, for a real-time collision detection using trajectory information from videos is presented in [71]. These attempts pave the way for exploring faster machine learning architectures for making swifter predictions for safety critical applications such as collision warning. In our work, we propose a ResNet-18 based collision prediction method that only uses entirely independent and ego-centric measurements obtained from inexpensive and lightweight FMCW radar and IMU sensors. Thus, our method is an infrastructure-free, self-sufficient solution that operates in cluttered environment and is faster and more accurate than traditional parametric and other learning-based methods.

5.4 Online Learning and Radars

Finally,learning-based collision prediction models should use self-labelled measurements from recent history and continue to learn. Therefore, a related field to explore is that of online learning, which has seen significant research in building algorithms that continuously and efficiently learn new information while retaining previously learned information. Most recently, rehearsal-based approaches for incremental learning have been implemented in which a model maintains a subset of previous examples that are mixed with new samples to update the model [23, 34, 57]. More specifically for radars, in order to operate in changing environmental conditions, cognitive or fully adaptive radars (FAR) can provide a feedback and optimization mechanism for improved system performance through the subsequent measurements [9, 24]. Another approach is of drift detection with incremental learning, where modification to the learning model occurs only when a change is detected in the environment distribution from which data are drawn, as is presented through a simulation study in [1]. In [74], an SVM-based method of working with hand-crafted features of the environment is presented that extends its underlying model by adjusting its hidden layer and retraining on selected incorrect samples while retaining previously learned information. Incremental learning, thus, is further explored in our work as the framework for our system to continuously learn with the changing environment using samples with the highest uncertainties.

6.1 Limitations of the Experiments Conducted

First, we note that the experiments in our work could be extended. For example, our experiments utilize only single-receiver radar measurements. In contrast, the use of multi-antenna radar systems that can beamform to improve the strength of the desired signal by combining signals from multiple antennas, adds the ability to measure angle and reduces the impact of interference. However, it adds to the processing and computational costs and has not been explored in this work. Also, our experiments use 24 GHz radar, and future experiments could explore performance in the 60 and 77 GHz bands which allow higher bandwidth and thus greater resolution. Finally, the speed of our moving prototype node is not representative of the target application of our system, such as sports players. However, radars are capable of capturing different speeds, according to the requirements by the application and radar operation settings, and extended experiments can include different speeds into the measurements.

6.2 Limitations of the Proposed Method

A collision prediction system for a particular application must have classification error rates such that the application needs are met. The proposed system, as presented in this paper, may not meet the needs of all applications. Future research is critical to advance collision prediction methods to meet the real-world usability standards in particular applications.

6.3 Future Work

Recurrent neural networks (RNNs) utilize the sequential nature of the data by using a time-series of multiple data points to make one inference. In our system, the data points are RDMs, and they are indeed sequential in time, being produced one after the previous. Using multiple sequential RDMs as one input to an RNN-based model may have value to explore in order to get improved performance compared to a CNN-based model. As a future direction to explore, other deep-learning architectures such as RNNs and recent modifications in CNNs that have shown to offer a lower complexity and inference latency, can be studied to present a comparative analysis of using raw radar data in predicting collisions. Sensor fusion with other modalities, such as video and lidar, can also be used to provide complementary information that could dramatically improve the classification rates of collision prediction. These methods may be beneficial in improving and increasing a machine learning model’s inference accuracy.