A clustering effectiveness measurement model based on merging similar clusters

AP algorithm

Frey & Dueck (2007) proposed the AP algorithm for clustering in SCIENCE in 2007. The main rationale behind the proposal of this clustering algorithm was to involve all data points as potential cluster centers and connect them to form a network. During its implementation, the cluster center of each data point is calculated through the transmission of information along each edge in the network (Wang, Chang & Du, 2021). The basic structure of the AP clustering algorithm is given as follows:

First, the similarity s between sample pairs is determined based on the Euclidean distance formula, and the similarity matrix S for all samples is obtained. Next, the availability a, and responsibility r of the samples are iteratively updated. When the number of iterative updates exceeds the preset value or the updating of representative points ceases after multiple iterations, the AP clustering algorithm terminates. At this point, the remaining samples can be assigned to the appropriate clusters to complete the clustering process. The operation process of the AP clustering algorithm is detailed below.

Step 1: Use the opposite number of the Euclidean distance as the similarity s (i,k) between the samples x_i and x_k to obtain the similarity matrix S. (1)${\displaystyle s\left(i,k\right)=\left\{\begin{array}{cc}-∥x_i-x_k∥\hfill & i\ne k\hfill \\ p\left(k\right)\hfill & i=k\hfill \end{array}\right.}$

where p (k) is the value on the diagonal of S that expresses the tendency of the sample x_k to be selected as the cluster center. The larger the p (k) value, the greater the probability of the sample x_k being selected as the representative of cluster centers. In this paper, reference is made to the practice recommended in Halkidi & Vazirgiannis (2001), i.e., taking the median of the similarity matrix S as the default value of the bias parameter p when no prior knowledge is available.

This article involves two key parameters bias parameter and damping parameter. A damping factor between 0 and 1 is applied to control the numerical oscillation. The data points in a sample have the same possibility for becoming the clustering center and the parametric value of all data points is set to the same value of P. Certain parametric values of the noise level in data, convergence criteria, affinity matrix construction, and similarity matric could affect the proposed approach.

The parameter selection for the AP algorithm is based on a combination of domain knowledge and empirical experimentation. The outcome of the proposed approach can be affected by bias and damping parametric values. For example, a higher preference value encourages more data points to become exemplars and a lower bias value makes it more challenging and results in fewer and larger clusters. Similarly, a lower damping factor value makes faster convergence but can result in instability or oscillation in the cluster assignments.

Step 2: Transmit information, iteratively update the availability a, and responsibility r, and generate a representative cluster center. The responsibility r(i,k) represents the degree of responsibility of x_k to x_i. The larger the value of r(i,k), the greater the probability of x_k becoming the center of x_i’s cluster. The availability a(i,k) represents the degree of availability of x_i to x_k. The larger the value of a(i,k), the higher the possibility of x_i’s cluster choosing x_k as its center. The methods for iteratively updating the responsibility r and availability a are given in Eqs. (2) and (3). (2)${\displaystyle r\left(i,k\right)=s\left(i,k\right)-\underset{k^{\prime }\ne k}{max}\left\{a\left(i,k^{\prime }\right)+\right.\left.s\left(i,k^{\prime }\right)\right\}}$

where a(i,k′) denotes the degree of availability of samples other than x_k to x_i; s(i,k′) denotes the degree of responsibility of samples other than x_k to x_i, i.e., the degree of completion of all samples other than x_k for the ownership of x_i; r(i,k) is the cumulative evidence proving that x_k has become the center of x_i. (3)${\displaystyle a\left(i,k\right)=\left\{\begin{array}{cc}\hfill min\left\{0,r\left(k,k\right)+\sum _{i^{\prime }⁄\in \left\{i,k\right\}}max\left[0,r\left(i^{\prime },k\right)\right]\right\}\hfill & \hfill i\ne k\hfill \\ \hfill a\left(k,k\right)=\sum _{i^{\prime }\ne k}max\left[0,r\left(i^{\prime },k\right)\right]\hfill & \hfill i=k\hfill \end{array}\right.}$

where r(k,k) denotes the self-responsibility of x_k, a(k,k) denotes the self-availability of x_k; r(i′,k) denotes the similarity level of x_k serving as the centroids of all samples other than x_i; a(i,k) represents the degree of possibility of taking all availability values greater than or equal to 0 plus the self-availability value of x_k as the cluster center.

Oscillations can occur when the responsibility r and availability a are updated. The damping parameter λ is introduced to reduce the amplitude of oscillation, eliminate oscillations, and correct r(i,k) and a(i,k) during iterations to make the iterative process more stable. The damping parameter is set to λ ∈[0, 1), and the number of iterations is t. The corrected iterative processes are expressed by Eqs. (4) and (5).

(4)${\displaystyle r{\left(i,k\right)}^{t+1}=\left(1-\lambda \right)\times r{\left(i,k\right)}^{t+1}+\lambda \times r{\left(i,k\right)}^t}$ (5)${\displaystyle a{\left(i,k\right)}^{t+1}=\left(1-\lambda \right)\times a{\left(i,k\right)}^{t+1}+\lambda \times a{\left(i,k\right)}^t}$

where r(i,k)^t and r(i,k)^t+1 denote the degrees of responsibility for the t^th and t +1^th iterations, and a(i,k)^t and a(i,k)^t+1 denote the degrees of availability for the t^th and t +1^th iterations.

Step 3: Determine the cluster-center representative. Select the sample x_k with the largest sum of responsibility r(i,k) and availability a(i,k) as the representative point of the cluster to which x_i belongs. The condition that the number of clusters k should satisfy is given in Eq. (6). (6)${\displaystyle k=\mathrm{arg max}\left\{a\left(i,k\right)+\right.\left.r\left(i,k\right)\right\}.}$

From the operation process described above, it can be seen that the bias parameter p appears when the responsibility r(i,k) is calculated in Step 1. As the value of p increases, r(i,k) and a(i,k) increase, and the probability of the candidate representative points becoming cluster centers also increases accordingly. It can be known that when the value of p is large and there are a large number of candidate representative points, the tendency of more candidate representative points to be cluster centers will become increasingly obvious. However, the current theoretical basis for determining the value of p is inadequate, which leads to the fact that the AP clustering algorithm usually takes the locally optimal solution or the approximate global optimum as its final result (Zhou et al., 2021). The practice of taking the median of similarity values as the value of p has a certain reference value (Li et al., 2017). Still, the resulting value of K is often greater than the correct number of clusters, and the clustering accuracy of this method appears insufficient. Therefore, it is necessary to further analyze and evaluate the clustering quality using appropriate evaluation indices to achieve more ideal results. In addition, when the sample size is huge, the AP clustering algorithm will experience problems such as insufficient storage space and long running time, and when it is used to process non-cluster-shaped sample sets, it will usually produce a large number of local clusters. Depending on a locally linear embedding (LLE) hybrid kernel function, an innovative AP clustering method was proposed in Sun et al. (2018), In addition to it, a new hybrid kernel was introduced to measure similarity and construct the similarity matrix for AP clustering. However, when the values of upper bounds for clustering obtained by this AP clustering method are large, this method will still face the problem of long running time. In a study Gan, Xiuhong & Xiaohui (2015), the merge process was integrated into the AP clustering algorithm to merge the two clusters satisfying the condition that the minimum inter-cluster distance is less than the average distance between clusters in the entire dataset, thus improving the performance of the proposed algorithm and solving the problem of unsatisfactory clustering results with non-cluster-shaped datasets. However, because this algorithm uses the minimum inter-cluster distance, sample data with low similarity may be assigned to one cluster in the merge process.

Internal evaluation indices

Internal evaluation indices have been widely used to perform the evaluation of the clustering quality for data without classification labels by measuring the intra-cluster and inter-cluster similarities after clustering using only the attributes of the dataset in question, and no external information is needed in such an evaluation process. High-quality clustering can achieve high intra-cluster compactness and good inter-cluster separation. These indices are usually achieved through certain forms of combination based on the selection of values for intra-cluster and inter-cluster distances (by taking extreme values or by weighting after taking extreme values). Commonly used internal evaluation indices and their characteristics are analyzed below.

Davies–Bouldin index

where K represents the number of clusters, C_i denotes the i-th cluster, n_i is the number of samples of the i-th cluster, v_i refers to the cluster center of the i-th cluster, d(x, v_i) is the Euclidean distance between the sample x and the cluster center v_i, and d(v_i, v_j) represents the Euclidean distance between the cluster centers v_i and v_j.

The Davies–Bouldin (DB) index of a sample set is obtained by taking the sum of the average distances between the samples in two adjacent clusters and the centers of the two clusters as the intra-cluster distance (Davies & Bouldin, 1979). The smaller the value of the DB index, the lower the similarity between clusters and the better the clustering quality. This index is suitable for evaluating datasets that are characterized by high intra-cluster compactness and great inter-cluster distance, but when the degree of inter-cluster overlap in a dataset is high (for example, when data points are distributed in a circular pattern), it is very difficult to complete clustering evaluation accurately using this index.

Xie-Beni index

where N represents the total number of samples, and u_ij refers to the fuzzy membership degree of sample x_j and cluster C_i.

In the Xie-Beni (XB) index, the numerator represents the intra-cluster compactness for fuzzy partitioning, and the denominator represents inter-cluster separation (Xie & Beni, 1991). This index takes into account the geometric structure of the dataset to be processed, calculates the intra-cluster correlation, and considers the inter-cluster distance. To achieve the best clustering result, the difference in the same cluster must be very small. The greater the value of the numerator, the higher the degree of inter-cluster separation. However, when the number of clusters is excessively large, the numerator of the XB index gradually decreases with the increase in K. As the value of K increases, the numerator of the XB index tends to 0, the value of the denominator increases continuously, and the XB index will lose the ability to judge due to its excessive monotony.

V_.P.C. and V_.P.E.

(9)${\displaystyle V_{PC}=\frac{1}{N}\sum _{i=1}^K\sum _{j=1}^N{u}_{ij}^2}$ (10)${\displaystyle V_{PE}=-\frac{1}{N}\sum _{i=1}^K\sum _{j=1}^N{u}_{ij}\times log{u}_{ij}.}$

Bezdek proposed two evaluation indices: partition coefficient (V_PC) and partition entropy (V_PE) (Bezdek, 1974b; Bezdek, 1974a). For both indices, the availability u_ij is used as the main criterion for evaluating the clustering quality. For V_PC, the closer the samples in a cluster are to the cluster center, the closer the value of u_ij is to 1. Therefore, the greater the value of V_PC, the better the clustering quality. V_PE is opposite to V_PC. The maximum value of u_ij is 1, and the value of u_ij will become 0 after the log is taken. For this reason, the value of V_PE will not be negative. Therefore, the smaller the value of V_PE, the better. The disadvantage of V_PC and V_PE is that they do not consider the spatial structure of clusters, and they are susceptible to the effects of noise and overlap between clusters.

VH&H

(11)${\displaystyle V_{H\&H}=\frac{\sum _{i=1}^K\sum _{j=1}^N{u}_{ij}^m{d}^2\left(x_j,v_i\right)}{N\times \left|{\mathrm{logmin}}_{s\ne t}{d}^2\left(v_s,v_t\right)\times w\right|}}$ (12)${\displaystyle w=\frac{p_s+p_t}{p_e}=\frac{\sum _{x_j\in C_s}{u}_{sj}\times log{u}_{sj}+\sum _{x_l\in C_t}{u}_{tl}\times log{u}_{tl}}{\sum _{i=1}^K\sum _{j=1}^N{u}_{ij}\times log{u}_{ij}}.}$

Lin, Huang & Huang (2015) improved the XB index from the perspective of entropy and proposed the index V_H&H. The numerator of V_H&H represents the degree of compactness, and the denominator represents the degree of separation. v_s and v_k denote the shortest distance between cluster centers. It can be seen that, for the degree of separation in this index, the log is taken to reduce the influence of the shortest distance between cluster centers. Because a negative value may occur after the log is taken, it is necessary to take an absolute value. Unlike the XB index, V_H&H introduces the concept of entropy into the degree of separation. According to this characteristic, the w in Eq. (12) is used as the weight of the degree of separation. This weight is the result of summing the entropies p_s and p_k in the two clusters with the shortest distance between their centers.

x_j and x_l are samples belonging to clusters C_s and C_k, and u_sj and u_kl denote the degrees of availability of x_j and x_l to C_s and C_k. When the clusters C_s and C_k with the shortest distance between their centers overlap each other, the entropies p_s and p_k of the two clusters will be high, i.e., the weight w will be large. Therefore, the degree of separation between the two clusters can be higher. To prevent either p_s or p_k from affecting the final calculation result excessively, both p_s and p_k need to be normalized. V_H&H has a high tolerance to overlapping between clusters, but it cannot evaluate samples with non-uniform densities stably.

Improved AP clustering algorithm

This study adopts the merging idea as described in a study Gan, Xiuhong & Xiaohui (2015). First, the AP clustering algorithm is used to perform the rough clustering of samples. Then, the initial clusters are merged based on similarity, and the upper limit of the number of clusters is reduced to compress the range of clusters and improve clustering accuracy. The general idea is to, on the basis of initial clustering performed by the AP clustering algorithm, calculate the ratio α of the similarity between any two clusters to the average similarity between clusters in the entire sample set. α represents the relationship between any two clusters and the entire sample set in terms of structural similarity. The smaller the value of α is, the closer any two clusters are to each other. If the lowest value of α is within the specified threshold range after complete traversals, the two clusters will be merged. If not, they will endure unmodified. The definitions and equations of the new algorithm are given below.

Definition 1 The Euclidean distance between any two points in space can be defined as: (13)${\displaystyle d\left(x_i,x_j\right)=\sqrt{\sum _{p=1}^l{\left(x_i^p-x_j^p\right)}^2}}$

where i =1,2, …, N; j =1,2, …, N; l represents the feature dimensionality of samples.

Definition 2 The similarity between any two clusters can be defined as taking the sum of the distances between all sample pairs in the two clusters. (14)${\displaystyle Sim\left(C_i,C_j\right)=\sum _{x_t\in C_i,x_u\in C_j,i\ne j}d\left(x_t,x_u\right)}$

where x_t and x_u denote any samples in the i^th and j^th clusters, respectively.

Definition 3 The average similarity between clusters in a sample set can be defined as the average similarity between two clusters in the entire sample set. (15)${\displaystyle \overline{Sim\left(X\right)}=\frac{\sum _{i=1,j=1,i\ne j}^K\left(Sim\left(C_i,C_j\right)\right)}{C_K^2}}$

where C²_K represents the number of combinations of any two clusters arbitrarily selected from K clusters.

Definition 4 Inter-cluster similarity ratio can be defined as the ratio of the similarity between any two clusters to the average similarity between clusters in the entire sample set.

(16)${\displaystyle {\alpha }_{i,j}=Sim\left(C_i,C_j\right)/\overline{Sim\left(X\right)}}$ (17)${\displaystyle C_i=\left\{\begin{array}{cc}{C}_i\cup C_j\hfill & if\left({\alpha }_{i,j}\le \mathrm{W}\right)\hfill \\ C_i\hfill & otherwise\hfill \end{array}\right..}$

If the inter-cluster similarity ratio α_ij is less than or equal to a given threshold W, the i^th and j^th clusters will be merged. Otherwise, they will remain unchanged. The reference range of W used in this paper is [0.3, 0.5]. The specific value of W can be set by users.

New CVIs

Intra-cluster and inter-cluster similarities are the main elements of the evaluation. Optimal clustering minimizes inter-cluster similarity while maximizing intra-cluster similarity. In this paper, the product of inter-cluster relative density and intra-cluster compactness is used to represent intra-cluster cohesion. The inter-cluster separation is represented by the ratio of the shortest distance between two clusters to the product of the distance between cluster centers and the inter-cluster overlap coefficient.

Halkidi & Vazirgiannis (2001) mentioned several indices for the evaluation of clustering algorithms. However, our current study identifies five indices that are particularly suitable for evaluating the Affinity Propagation (AP) algorithm. These indices have demonstrated superior performance in similar studies. For instance, the author stated that intra-cluster cohesion is best suited for unsupervised clustering algorithms (Estiri, Omran & Murphy, 2018). The selection of a small set of indices allows for a more focused and in-depth analysis of clustering performance. This enables us to conduct a targeted evaluation of the clustering algorithm’s performance, aligning with our primary interest in the current research.

The definitions and equations of the new indices are given below.

Inter-cluster dispersion

To minimize the similarity between clusters, the data points in different clusters should be kept as far away from each other as possible. In this paper, the ratio of the shortest distance between two clusters to the product of the distance between cluster centers and the inter-cluster overlap coefficient (hereinafter referred to as overlap) is used to represent inter-cluster dispersion (hereinafter referred to as disp). Inter-cluster dispersion is defined as follows: (23)${\displaystyle Disp\left(i,j\right)=\frac{mind\left(x_i,x_j\right)}{d\left(v_i,v_j\right)\times overla{p}_{ij}}}$

where x_i and x_j denote any samples in clusters C_i and C_j, and v_i and v_j denote the centers of the two clusters. The greater the value of this ratio, the more dispersed the clusters, the clearer the cluster boundaries, and the better the clustering quality. In the equation above, the overlap coefficient overlap _ij represents the degree of concentration of data points in the overlapping area between clusters C_i and C_j. The overlap coefficient has several practical applications in a range of research areas. For example, in text document clustering, documents belong to several multiple topics. The overlap coefficient metric is used to analyze the extent to which topics overlap, and potentially reveal the areas of semantic ambiguity. Similarly, the level of ambiguity in object recognition is determined by using the overlap coefficient metric in image segmentation (Zou et al., 2004). The overlap coefficient is used in the fractional segmentation of MR images for reproducibility and accuracy.

The overlap coefficient is defined as follows:

(24)${\displaystyle overla{p}_{ij}=1+\frac{\sum _{x_a\in C_i\cup C_j}f\left(x_a,C_i,C_j\right)}{n_i+n_j}}$ (25)${\displaystyle f\left(x_a,C_i,C_j\right)=\left\{\begin{array}{cc}{e}^{1-\left|u_{ia}-u_{ja}\right|}\hfill & if\left|u_{ia}-u_{ja}\right|\le \mathrm{H}\hfill \\ 0\hfill & otherwise\hfill \end{array}\right.}$ (26)${\displaystyle u_{ia}=\frac{1}{\sum _{m=1}^K{\left(d\left(x_a,v_i\right)/d\left(x_a,v_m\right)\right)}^2}}$ (27)${\displaystyle u_{ja}=\frac{1}{\sum _{p=m}^K{\left(d\left(x_a,v_j\right)/d\left(x_a,v_p\right)\right)}^2}}$

where x_a denotes the data in the cluster C_i or cluster C_j; n_i and n_j denote the number of data points in C_i and the number of data points in C_j; f(x_a, C_i, C_j) represents the contribution of the sample x_a to the degree of concentration in the overlapping area between clusters C_i and C_j; u_ia and u_ja are the degrees of availability of x_a to clusters C_i and C_j (Dunnâ, 1974); and H is the threshold at which x_a falls within the overlapping area. The default value of H used in this paper is 0.3. As shown in Fig. 1, x_b and x_c are located inside C_i and C_j, respectively, x_b ∈C_i, and x_c ∈ C_j. The availability of x_b and x_c is very clear, i.e., —u_ib- u_jb— >H, — u_ic- u_jc— >H. Compared with the availability of x_b and x_c, the availability of the data point x_a located in the overlapping area between the two clusters (as shown in Fig. 1) is closer to u_ia and u_ja, i.e., —u_ia- u_ja— ≤H. f(x_a, C_i, C_j) determines the inter-cluster overlap coefficient by calculating the degree of concentration of fuzzy data points in the overlapping area.

Specifically, when —u_ia- u_ja— ≤ H, data points will fall within the overlapping area. In this case, e^{1−|u_ia−u_ja|} is taken to increase the influence of data points in the overlapping area. Otherwise, zero is taken. From Eqs. (24) and (25), it can be known that the smaller the cumulative sum of the degrees of concentration in the overlapping areas between clusters, the smaller the overlap coefficient, and the clearer the division of clusters.

Clustering effectiveness measurement model based on merging similar clusters

In this article, the original AP clustering algorithm is improved by merging similar clusters, and a clustering effectiveness measurement model based on merging similar clusters (hereinafter referred to as the CEMS) is proposed based on the new evaluation indices presented herein. The structure of the CEMS is shown in Fig. 2.

The operation process of the CEMS consists of 10 steps intended to merge similar clusters and determine the optimal number of clusters. These steps are described below.

(1) Calculate the similarity matrix S of dataset X using Eq. (1).

(2) Calculate and update the responsibility and availability using Eqs. (2) and (3), set the damping parameter and the number of iterations, and use Eqs. (4) and (5) to correct the responsibility and availability during iterations and reduce the amplitude of oscillation.

(3) Select the data point with the maximum sum of responsibility r and availability a according to Eq. (6), set the selected data point as the representative point of the cluster to which it belongs, repeat Step (2), obtain the representative points of all clusters, and complete the clustering of non-representative points based on similarity.

(4) Calculate the similarity between any two clusters and the average similarity between clusters in the entire sample set using Eqs. (14) and (15).

(5) Calculate the inter-cluster similarity ratio using Eq. (16) and use Eq. (17) to merge clusters that satisfy the required condition.

(6) Repeat steps (4) and (5), update clusters based on the new inter-cluster similarity ratio, and obtain the maximum number of clusters K_max at the end of the iterative process.

(7) Take K_max as the initial value of the parameter K in the CVI, i.e., K = K_max.

(8) Calculate inter-cluster relative density (den), intra-cluster compactness (com), and inter-cluster overlap coefficient (overlap) using Eqs. (19), (20), and (24), calculate intra-cluster cohesion (coh) and inter-cluster dispersion (disp) using Eqs. (18) and (23), and calculate the CVI using Eq. (28).

(9) Let K =K-1 and repeat steps (7) and (8) until K = 2 to obtain a set of CVI values, i.e., CVI _K = {CVI (2), CVI (3),…, CVI (K_max-1), CVI (K_max)}.

(10) Select the parameter K corresponding to the maximum value of CVI according to Eq. (29), and output it as the optimal number of clusters K_opt.

CEMS effectiveness test

In this section, a comparative experiment on the maximum number of clusters for the UCI dataset was conducted using the improved AP clustering algorithm and the original AP clustering algorithm. The settings of these two algorithms are as follows: the damping factor λ = 0.85, the number of iterations t = 1,500, and the threshold of inter-cluster similarity ratio for the improved AP clustering algorithm W = 0.3. The results are given in Table 2. The clustering results were evaluated using the CVI and classical internal evaluation indices. The comparison results are summarized in Table 3.

As shown in Table 2, the values of upper bounds for the improved AP clustering algorithm on the UCI dataset are all smaller than those for the original AP clustering algorithm, indicating that the former can effectively compress the clustering space.

As shown in Table 3, the accuracy of the CVI concerning the number of clusters is 83.33%, and the accuracy levels of other indices are 33.33%, 16.67%, 33.33%, 50%, and 66.67%, respectively. It is obvious that the accuracy of the CVI is significantly higher than the accuracy levels of the other five indices. It is to be noted that during the test on the Glass dataset, all indices except the CVI failed to achieve accurate classification. During the test on the Dermatology dataset, all indices failed to achieve accurate classification, and the CVI divided the data points in this dataset into five clusters. Compared with the number of clusters determined by other indices, the number of clusters determined by the CVI is closer to the standard number of clusters. In general, the CVI performs best on this dataset. It indicated that CVI better performed on the high dimensional datasets due to the thorough exploitation of spectral dimension reduction, optimization of data architecture, and reduction in the overlapping between various clusters. As a result of it, the clustering performance is improved.

Application of the CEMS in intrusion detection

To test the performance of the CEMS, 20% of the KDDTrain+ samples in the NSL-KDD intrusion detection dataset were experimentally selected to create a training set with 25,196 samples, and 50% of the KDDTest+ samples were selected and divided into three test sets with 11,272 samples in total. The samples in the training and test sets contain four network intrusions, namely, Dos, Probe, R2L, and U2R (Chen & Wang, 2022). The formats of the samples were converted by one-hot encoding (Zhang et al., 2019), and the feature dimensionality of the datasets was reduced using the dimensionality reduction method. The details of pre-processed data are listed in Table 4.

Relationship between the CVI and K

The improved AP clustering algorithm was run with the parameters listed above. The maximum number of clusters in training is set as K_max = 28, denoted along the x-axis. The CVI values corresponding to different values of K are shown in Fig. 3, denoted along the y-axis. When 2 ≤K ≤20, the CVI value increases continuously. When K ∈ as mentioned in Li et al. (2017), the increase in the CVI value slows down. When K = 24, the CVI value reaches its peak.

As the value of K continues to increase, the CVI value decreases slowly. When K = 27, the rate of decrease in the CVI value increases. Considering the rules of variation in the CVI, the set of multiple numbers of clusters corresponding to the process of the CVI slowly increasing to the maximum value and then slowly decreasing from the maximum value is defined as the optimal clustering space denoted as K_opt ∈ as mentioned in Davies & Bouldin (1979).

Indices for intrusion detection in the optimal clustering space

In this section, the effectiveness of the indices for intrusion detection in the optimal clustering space K_opt ∈ was verified using three test sets (Davies & Bouldin, 1979).

Table 5 presents various indices for three test sets (T1–T3). Clusters 20–26 are provided with their corresponding detection rate, correct classification rate, false negative rate (FNR), and false positive rate (FPR) values. From the values listed in Table 5 and the line charts in Fig. 4, it can be seen that when K = 22, the detection rates and FNR of the three test sets reach extreme values at the same time, and the average detection rate and FNR are 94.13% and 4.76%, respectively; when 21 ≤ K ≤ 23, the correct classification rates of the test sets reach the maximum values successively, with an average of 93.40%; when K = 21 and K = 22, the FPR of the three test sets reach the minimum values, with an average of 4.36%.

Time complexity analysis

The AP algorithm exhibits quadratic complexity, meaning that as the size of a dataset or the number of data points increases, the computational demand of the AP algorithm grows quadratically. Performance peaks of the algorithm were observed at a specific cluster (K = 22), indicating that as the number of clusters increases, the computational requirements of the AP algorithm also escalate. Therefore, due to the quadratic time complexity of the AP algorithm, it is crucial to carefully consider its application, especially with large datasets.

Comparison between the CEMS and other algorithms

The performance comparison of CEMS and other algorithms is presented in Tables 6 to 9. Specifically, Table 6 showcases the comparison of detection rates (Mohammadi et al., 2019; Su et al., 2020), while Table 7 displays the comparison of correct classification rates (Zou & Yang, 2018a; Zou & Yang, 2018b).

For further verification of the CVI and compare the clustering quality of the CEMS and that of other algorithms horizontally, the K-medoids clustering algorithm and two improved density-based clustering algorithms (Zou & Yang, 2018a; Zou & Yang, 2018b) were combined with the CVI and recorded as “K-medoids+,” “Reference (Zou & Yang, 2018a)+,” and “Reference (Zou & Yang, 2018b)+.” Four intrusion detection indices were horizontally compared within the range of K_opt ∈ as mentioned in Davies & Bouldin (1979). The k-medoids algorithm attempts to minimize the distance between two points and its centroid. For the K-medoids+ algorithm, the average value was taken after it was run 500 times.

By observing the data in Tables 6 and 7, it can be found that, after the other three algorithms are combined with the CVI, the detection rate, as well as the correct classification rate, can receive their highest values when 21 ≤ K ≤ 23, and the detection rate of the CEMS is significantly higher than those of the other three algorithms within the whole range given above. The correct classification rate of the CEMS is consistent with that of the algorithm proposed in Zou & Yang (2018b) when K = 23 and the CEMS outperforms the other three algorithms in the ranges of 20 ≤K 23 and 25x K ≤ 26. As shown in Tables 6 and 7 the proposed CEMS achieved the best performance in terms of detection rate and accuracy (%) on the NSL-KDD dataset. Results clearly show that the detection and classification performances of the proposed CEMS are more effective compared with the rest of the clustering algorithms.

From the data in Table 8, it can be seen that the FNR of the other three clustering algorithms can reach the minimum values when 21 ≤ K ≤ 23, and the minimum FNR of the CEMS is 4.76%, which is slightly lower than that of “Reference Zou & Yang (2018b)+” and significantly lower than those of the other two algorithms. The performance of the CEMS is better compared with the K-medoid, and clustering models in Reference Zou & Yang (2018a) and Reference Zou & Yang (2018b). False negatives are either due to poor detection probabilities or failure in connecting clusters that are too far apart. However, in the current study, the proposed CEMS’s reduced FNR indicates its robust and efficacious performance.

By observing the data in Table 9, it can be known that the FPR of the other three algorithms can reach the minimum values when 21 ≤ K ≤ 22; when 20 ≤ K ≤ 23, the FPR of the CEMS is slightly lower than those of the other three algorithms; and when 24 ≤ K ≤ 26, the average FPR of the CEMS is basically consistent with those of “K-medoids+” and “Reference Zou & Yang (2018a)+” and slightly higher than that of “Reference Zou & Yang (2018b)+”.

The experimental results above show that the CVI has a high universality and can accurately assess the results from clustering and give an optimal number of clusters after it is combined with other clustering algorithms. During the comparative test, the CEMS could improve the intrusion detection rate and correct classification rate at a low FPR and significantly reduce the FNR, indicating that its overall performance is more desirable than that of the other three clustering algorithms.

The current research shows promising results on both the UCI and NSL-KDD datasets. This suggests that the findings of this study may be generalized to other datasets with similar characteristics. However, the applicability of the proposed research may be limited if new datasets have different features.

This research show cases correct classification and clustering, demonstrating that the proposed approach was effectively applied. The use of internal evaluation indices has further strengthened confidence in the internal validity of this study. It demonstrates that the observed effects and their association are genuinely attributed to the performance of the CEMS algorithm. This implies that the proposed research was well-designed and conducted with minimized potential for biases in the experimental design.