Application of artificially intelligent systems for the identification of discrete fossiliferous levels

Unsupervised machine learning

For initial detection of hidden patterns among the levels of Batallones Butte site, an unsupervised density based clustering algorithm was used. Unsupervised ML algorithms are highly efficient AIAs that are exceptional for their use in tasks such as pattern recognition, anomaly detection, noise and dimensionality reduction, feature engineering as well as generative modeling (Patel, 2019). While unsupervised learning performs poorly in specifically defined tasks, their greatest advantage can be seen through their flexibility when applied to versatile datasets for general feature extraction. A popular application of unsupervised algorithms lies in clustering tasks, a component of pattern recognition that is, considered useful for processing and searching for hidden trends in highly noisy datasets.

A non-parametric Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm from the “fpc” R package was used for pattern recognition in 3D spatial data. Proposed as a means of overcoming clustering issues where groupings of samples are not straightforward (Ester et al., 1996), DBSCAN is highly efficient at detecting patterns in arbitrary and noisily distributed samples (Sander et al., 1998; Schubert et al., 2017).

DBSCAN performs through establishing areas of significant point densities, thus providing an efficient means of finding non-parametric patterns among noisy data sets while easily detecting outliers in low-density regions. This is especially useful when used for pattern recognition in spatial studies, where traditional clustering models such as partitioning and hierarchical algorithms tend to focus groupings around a centroid or subsamples in a convex manner. This frequently creates conflict in real-life circumstances.

DBSCAN has proven efficient when applied to 2D, 3D and/or higher dimensional feature spaces (Ester et al., 1996; Sander et al., 1998; Schubert et al., 2017). The algorithm works by separating points within the cluster (i.e., core points) from those on the border of the cluster (i.e., border points). The differentiation between the two is established mathematically depending on the reachability of neighboring points (q) from a core point (p). To define this, DBSCAN requires two main hyperparameters, the ε value defining the neighborhood (a.k.a. Eps) of a point and the minimum number of points (MinPts) that are required to form the cluster (i.e., the density of points). From here DBSCAN defines the neighborhood of p within dataset D as N_Eps(p) = {q ∈ D | dist(p, q) ≤ ε}, adjusting for border points and the exclusion of noise through p ∈ N_Eps(q) and |N_Eps(q)| ≥ MinPts (Ester et al., 1996). The definition of density-reachable points for clustering is thus established if the points can be linked in a chain, whereby p₁, …, p_n = q, p_n = p ensuring p_i+1 is directly reachable through density from p_i. Through this definition, border points may not be density reachable if they do not fulfill the core point condition |N_Eps(q)| ≥ MinPts. The final components that define point density for clustering are established through density-connected points, whereby a point o is density-reachable from both p and q, essentially connecting both border and cluster points.

Once density has been established within the sample, the assigning of these points into cluster C requires the fulfillment of conditions; (1) ∀ p, q: if p ∈ C and q fulfill the definition of being density reachable, then q ∈ C, while (2) ∀ p, q ∈ C: p must be density-connected to q. These defined conditions are named Maximality and Connectivity, respectively (Ester et al., 1996). In accordance with the aforementioned conditions, the final definition of noise can be logically established through {p ∈ D | ∀ i: p ∉ C_i}.

DBSCAN was thus trained on the entire dataset in an unsupervised manner, establishing an average MinPts value between 3 and 5. These values were intuitively chosen depending on slices with tightly packed densities (MinPts ≈ 5) and slices with a lower number of remains (MinPts ≈ 3). This abides by general rules recommended by numerous authors (Sander et al., 1998; Schubert et al., 2017), who define an approximate optimum MinPts = 2 × n° dimensions in the dataset which is then adjusted depending on the complexity of point distributions and the analyst’s knowledge of the domain under consideration (Sander et al., 1998; Schubert et al., 2017). ε values were established in accordance with both the correspondent MinPts values and the slices under study. For optimization of this hyperparameter, k-distance graphs were plotted according to the nearest neighbor (Ester et al., 1996), employing the “elbow” technique to find a ε value according to both the MinPts parameter and the actual dataset (Thorndike, 1953; Patel, 2019; Satopa et al., 2017). The use of this heuristic, as provided by the “dbscan” R package, can then be adjusted to find the smallest ε value possible that can define a final model for training. DBSCAN then establishes clusters within the dataset, separating noise into a separate “0” cluster and creating convex hulls through the use of border points q.

Collaborative intelligence learning

Considering the currently unquantifiable nature of numerous geological features that ML algorithms may be unable to detect, a human-in-the-loop collaborative strategy was proposed for this study (Kamar, 2016; Holzinger, 2016; Dellermann et al., 2019). The objectives of including this hybrid intelligence strategy meant that human interaction could complement the strengths of the pattern detection algorithms (Simard et al., 2017; Dellermann et al., 2019), whereby creating a bridge between both unsupervised and supervised techniques for model creation.

The Expert-in-the-Loop (EitL) approach adopted here consisted in the revision of the clustered dataset by considering possible underlying geological components that could be generating noise undetectable by the model. Components of this nature could easily include disconformities, geological faults, erosive surfaces, uneven sedimentation or even sedimentation of geological elements such as large boulders that could be separating point-densities that would essentially belong to the same layer.

The strategies employed consequently used EitL (in this case geologist-in-the-loop) interaction to manually correct clustered patterns, assessing which of the clustered groups were geologically separated and assigning these clusters to a new labeled layer. This was performed as objectively as possible through complimenting paleontological data routinely collected from the site during excavation and evaluating the nature of each of DBSCAN’s groupings. EitL thus benefits through Machine Teaching (MT) in as much as the expert knowledge is used for troubleshooting and debugging (Dellermann et al., 2019; also known as a sense-making approach). Moreover, the amount of human input for models was monitored in a collective manner, using numerous domain experts to ensure accuracy (Dellermann et al., 2019). In cases where clusters seemed dubious or could not be clearly defined by either the EitL participants or the MT algorithm, these were stripped of their labels and included in the noise (cluster “0”) group for objective classification by the trained ML algorithms in the supervised phase of the system. Once revised by the EitL participants, each cluster group was assigned a new label that could be used to train supervised algorithms in the following part of the workflow.

Supervised learning

Once spatial data points had been passed through the DBSCAN algorithm for initial pattern detection and then corrected by geologist-in-the-loop interventions, supervised algorithms were employed to define the final fossiliferous level models and classify those points in cluster group “0”.

Considering the amount of data available, no bootstrapping procedures were deemed necessary prior to supervised training. Each algorithm was thus trained using 70:30% [train:test] splits using k-fold cross-validation (k = 10) in order to ensure the model could efficiently adjust its weights. While some authors propose the use of Spatial Cross-Validation (SCV) for studies regarding geographic data (Miller, 2004; Brenning, 2012; Pohjankukka et al., 2017; Lovelace, Nowosad & Muenchow, 2019), these algorithms were not seen to produce any significant change to the quality of results. Additionally, hyperparameter optimization was performed using a random search loop function programed in R; establishing the best hyperparameter values for each model using a combination of random values until finding the optimum settings (Bergstra & Bengio, 2012). These loop algorithms ran for 50 iterations and were then extrapolated and used for the final classification models.

Two primary supervised ML algorithms were used for the final fine tuning of the fossiliferous level models.

• Support Vector Machines (SVM). SVMs map out input vectors into a non-linear high dimensional feature space, using hyperplanes to calculate the degree of separation between samples (Cortes & Vapnik, 1995). In order to overcome traditional limitations imposed by linearity, a kernel function is used to define the feature space (Bishop, 2006). The constructed hyperplane can thus be used as a discriminant classifier decision surface which uses maximized margin or decision boundaries to reduce chances of overfitting (Cortes & Vapnik, 1995; Bishop, 2006). The consequent hyperplane can then be used to plot the maximized margin separations between samples and can provide a visual means of dividing strata, in an efficient and objectively computed way. For SVM applications the “e1071” R package was used.

• Random Forest (RF). RF can be described as a robust and highly complex form of decision tree which has proven useful in the past for the processing of spatio-temporal data (Hengl et al., 2018). The RF algorithms use small random numbers of data set variables rather than the whole dataset, constructing an independent decision tree with each subsampling (Breiman et al., 1984; Breiman, 2001). The random variable selection is additionally performed using bootstrap aggregation, using a technique frequently referred to as out-of-bag observations. RF is then able to calculate the number of iterations needed to minimize the out-of-bag error. Once the number of trees has been selected, the algorithm averages results to produce a robust classification model (Breiman, 2001). For RF applications the “caret” R package was used.

Model evaluations were performed following standardized ML protocol, evaluating the Kappa (κ), Sensitivity, Specificity and Balanced Accuracy values obtained through confusion matrices (Kuhn & Johnson, 2013; Lantz, 2013). Sensitivity, Specificity and Balanced Accuracy are values derived from evaluations of Type I and Type II statistical errors in proportion with the rest of the calculated confusion matrix. These values are presented as numbers between 0 (poor) and 1 (high performing classifiers). κ values are a further statistical adjustment of the accuracy metric that measure model agreement relative to what would be expected by chance (Kuhn, 2008). These values are similarly presented as numbers between 0 and 1 although negative values can (although rarely) occur. Further evaluation of this statistic uses 0.8 as the threshold between poor and powerful classification models. Finally, loss metrics were recorded to evaluate the predictive power of each model. For this, the Mean Squared Error (MSE) metric was employed: $$\frac{1}{n}\sum _{i=1}^n{\left(t_i-p_i\right)}^2$$taking into consideration the target (t_i) and predicted (p_i) values for each classification. MSE values are further interpreted considering AIAs of this type are trained to reduce the error produced in CV sets, therefore the smaller the MSE value the more powerful the predictive model. Model evaluation was performed using the “caret” R package.

Fine tuning of fossiliferous levels

Once the optimal models were defined, these were used to classify any of the points separated as noise by DBSCAN as well as the clusters that the EitL participants could not objectively define as a part of either fossiliferous level (so as to avoid subjectivity). In order to empirically class any point as a member of a fossiliferous level, the trained SVM and RF algorithms were used to predict the class label of each point. A threshold of 80% security was established as acceptable. Any points that could not be classed by the model with over an 80% predictive decision boundary were thus rejected and therefore not classified into any level in the final fossiliferous levels models.

Unsupervised machine learning clustering

DBSCAN clustering proved highly efficient at quickly processing each slice, grouping fossil remains into abundant clusters in both Batallones-3 (Fig. 2) and Batallones-10 (Fig. 3). DBSCAN results in most cases present very noisy profiles, with the detection of large numbers of tightly packed clusters. This is a result of the irregular density patterns detected in the data and probably results from numerous geological and paleontological disconformities that the algorithm is unable to quantify in some areas. Nevertheless, profiles become less noisy towards the extremities of each slice, helping the differentiation between levels, identifying at least three different fossiliferous levels.

On average, DBSCAN identified 166 points (5.6%) as noise in Batallones-10 and 62 (7.76%) in Batallones-3, with the highest levels of noise appearing in Batallones-10 x axis slice (n = 141) and Batallones-3 right slice (n = 56). For Batallones-3 right slice, this can be attributed to certain areas of the profile having little separation between the layers (Fig. 2C). Nevertheless, excluding noise, most clustered groups seem to follow a trend or pattern that highlights a separation between different fossiliferous levels (see “Collaborative Intelligence Learning”).

Collaborative intelligence learning

While DBSCAN results produced a large number of clusters, detailed analysis through the EitL process identified numerous groupings that belonged to the same fossiliferous level. After careful evaluation, each of these separations could be recognized as products of minute or large disconformities. These can be attributed to the cavity’s asymmetrical geomorphology at Batallones-10, visible in the x axis slice (Fig. 4A) or to a fallen large carbonate block in Batallones-10, as seen in the y axis slice (Fig. 4B). This highlights the importance of the geologist-in-the-loop hybrid intelligence system for clarification and unification of clusters.

In other cases, numerous examples can be seen where depositional processes have placed fossiliferous levels closer together such as the sedimentary onlap between levels II and III in Batallones-10 x axis slice (Fig. 4A) and Batallones-3 right slice (Fig. 5B). Nevertheless, considering the algorithms’ effectiveness processing 3D data, DBSCAN is still able to detect density patterns across the slice’s x, y and z axes, thus separating the different levels.

Supervised learning

Once processed and cleaned by both DBSCAN and the EitL specialists, databases were channeled into supervised algorithms for training. In all cases, each of the databases generated by unsupervised learning algorithms proved to be highly efficient for the training of their supervised counterparts. Both SVM and RF obtained exceptional results with over 90% accuracy when differentiating between groups and small MSE values on all accounts (Table 1). While SVM proved to have greater balanced accuracy on training sets, RF in general obtained optimal MSE values in testing, especially in the case of Batallones-10. Interestingly Batallones-3 (both left and right slices) seems to create the most amount of confusion, nevertheless on all accounts κ values remain above the acceptable threshold.

Using AIAs to clean the final profiles and de-noise the datasets, both SVM and RF proved to be confident classifiers when making predictions. In most cases, over half the points denoted as noise by the previous phases were successfully classified and assigned into a palaeontostratigraphic level (Table 2). For Batallones-10 y axis slice and Batallones-3 left slice, RF outperformed SVM with greater MSE values and success in classifying indeterminable points. Nevertheless, in either case, RF tended to have greater decision making capabilities on some individual points than SVM, even though SVM obtained greater overall MSE in others.

Upon further evaluation of model performance, SVM maximized margin decision boundaries (dotted line in Fig. 6) are seen to expand or contract closer to the identified sedimentary onlap areas (in both Batallones-3 and Batallones-10) while an increase in MSE is observed for these particular areas as well (Table 2).

Fine-tuned models

Once the data had been processed by each of the phases of the workflow, a final fine-tuned model of the fossiliferous levels was created.

The fine-tuned model produced for Batallones-3 shows three discrete fossiliferous levels on either side (left and right) of the debris cone (Fig. 7). The left profile (Fig. 7A) is laterally less extensive (~3 m) than the right profile (Fig. 7B), which extends over 5 m. All identified levels in both profiles dip from the debris cone outwards, with dip angles decreasing towards the outermost parts of the cavity and towards the top of the infilling, with Level 3 dipping only slightly in Batallones-3 left profile (Fig. 7A) and nearly horizontal in the outer part of the Batallones-3 right profile (Fig. 7B).

Batallones-10 fine-tuned model shows three different discrete fossiliferous levels (Fig. 8). A first level (Level I), dips northward (Fig. 8A) and towards the East and West dips towards the central part of the cavity, adapting to the cave limits (Fig. 8B). Level II conformably overlies Level I in the southern section, dipping towards the North and in the northern, eastern and western cavity limits dips towards the central part of the cavity (Figs. 8A and 8B). Finally, Level III is practically horizontal, with gentle folding in the outermost limits of the cavity (Figs. 8A and 8B).