Two novel outlier detection approaches based on unsupervised possibilistic and fuzzy clustering

Studies on clustering-based outlier detection

Choosing an appropriate clustering algorithm for outlier detection is difficult and usually application dependent task because it depends on many criteria such as the size of data set, types, distribution and number of features, level of outliers in data, shapes of existing clusters, time constraints and many others (Ben-Gal, 2005; Penny & Jolliffe, 2001). Although there is a few outlier detection techniques based on the hierarchical clustering algorithms (Loureiro, Torgo & Soares, 2004), the partitioning algorithms have been widely used in outlier detection especially when the size and number of features are primary concerns. Among them, K-means has been the most widely used algorithm in many clustering-based outlier detection studies. On the other hand, as mentioned in many studies, K-means is sensitive to noise and outliers, and may not give accurate results (Hodge & Austin, 2004; Gan & Ng, 2017). Alternatively, K-medoids or partition around medoids (PAM) is less sensitive to local minima problem and, therefore, some studies targeted to use these hard clustering algorithms in outlier detection (Jayakumar & Thomas, 2013; Kumar, Kumar & Singh, 2013). However, the hard clustering algorithms such as K-means and PAM force each data point to belong to the nearest cluster. In this case, it becomes difficult to find outliers when they do not tend to form small or sparse clusters.

As a soft clustering algorithm, the Fuzzy C-means (FCM) algorithm calculates different membership degrees for each data point to every cluster. When a data point has the same degree of membership to the clusters in a partitioning task it can be evaluated as an outlier. For detection of outliers using fuzzy clustering algorithms, Klawonn & Rehm (2005) proposed a method that combines FCM with a modified version of Grubbs (1969), which is a statistical outlier detection test for univariate data. In their approach, the mean and standard deviation of each feature for each cluster is calculated after a clustering analysis, and then each feature is tested against a critical value. The feature vector with the largest distance to the mean vector is assumed to be an outlier, and removed from the data set. With the new data sets, the outlier tests are repeated until no outlier is found. The other clusters are processed in the same way.

Rehm, Klawonn & Kruse (2007) introduced an outlier detection method using noise clustering with FCM. Their approach determines the noise distance preserving the hyper-volume of the feature space when approximating the feature space using a specified number of prototype vectors. They obtained high accuracy with their approach for different numbers of clusters in FCM runs.

Moh’d Belal, Al-Dahoud & Yahya (2010) also used the FCM algorithm and proposed a method based on testing the difference of the objective function values. In their method, small clusters are determined as outlier clusters and removed from data set after the first run of FCM. The remaining outliers are then determined by computing the differences between objective function values by temporally removing the points from the analyzed data set.

Noise may be present in data sets due to random errors in a feature and they should not be considered as outliers. But they may distort the distribution and mask the distinction between normal objects and outliers. Regarding this kind of problems, the possibilistic algorithms are good candidates for distinguishing outliers more efficiently. For example, Treerattanapitak & Jaruskulchai (2011) stated that integrating the possibilistic and fuzzy terms in a clustering algorithm allows detecting outliers. In their study, possibilistic exponential fuzzy clustering produced accurate results in outlier detection based on exponential outlier factor scores that are calculated from the distances to the centroids.

Even though the main goal of clustering algorithms is to divide data set into homogenous clusters, in recent years, there is an increasing interest to extend them with some approaches for detecting outliers. Duan et al. (2009) proposed the CBOF, a clustering-based outlier detection algorithm requires four parameters. Huang et al. (2016) introduced the NOF algorithm, a non-parameter based on natural neighbor. Gan & Ng (2017) shortly reviewed the extended algorithms based on K-means and introduced the K-means with outlier removal (KMOR) algorithm which handles clusters and outliers simultaneously. Recently, Huang et al. (2017) proposed the ROCF, a cluster detection algorithm does not require a top-n parameter. ROCF detects isolated outliers and outlier clusters based upon a k-NN graph. The idea behind ROCF is that outlier clusters are smaller in size than normal clusters.

Although all the algorithms discussed above have their own advantages, they need to select at least one or more parameters. So, using the simple methods which do not require data-dependent parameters may be more useful in outlier detection. Although the typicality degrees from an optimal run of a possibilistic clustering algorithm can be used to identify outlying points, since the fuzzy and possibilistic algorithms are more robust to the noise and coincident clusters problems they can be more successful to detect the outliers. In this study, we used the unsupervised fuzzy possibilistic clustering algorithm by Wu et al. (2010) as the representative of fuzzy and possibilistic algorithms.

Unsupervised fuzzy and possibilistic clustering algorithms

The hard prototype-based clustering algorithms, i.e., K-means and its variants, assume that each object belongs to only one cluster; however, clusters may overlap and objects may belong to more than one cluster. In this case, a data point can be a member of several clusters with varying membership degrees between zero and one. The Fuzzy C-means (FCM) clustering algorithm (Bezdek, 1993) assigns a fuzzy membership degree to each data point based on their distances to the cluster centers. If a data point is closer to a cluster center, its membership degree to that cluster will be higher than its membership degrees to the other clusters.

To fix the sensitivity of FCM to noise and outliers, the possibilistic C-means (PCM) algorithm has been introduced (Krishnapuram & Keller, 1993). However, it has been revealed that PCM can generate coincident clusters if it is not well initialized. Later a dozen of hybrid versions of FCM and PCM have been proposed to overcome the problems with FCM and PCM. For instance, the fuzzy possibilistic C-means (FPCM) Pal, Pal & Bezdek (1997) aims to compute memberships and typicalities simultaneously. An extended version of PCM has been introduced by Timm et al. (2001) to make clusters far away from each other. The possibilistic fuzzy C-means (PFCM) Pal et al. (2005) has also been developed to fix the noise sensitivity problem with FCM, the coincident clusters problem with PCM and row sum constraints problem with FPCM.

Wu et al. (2010) proposed the unsupervised possibilistic and fuzzy clustering (UPFC) algorithm as an improved version of the original possibilistic clustering algorithm (PCA) (Yang & Wu, 2006). UPFC combines FCM and PCA to overcome the noise sensitivity problem of FCM and the coincident clusters generated by PCA. Moreover, unlike PCA algorithm, UPFC also does not require a fuzzy membership matrix from a previous FCM run. Using the objective function in Eq. (1), UPFC minimizes the distances between c prototype vectors (v_i) and n feature vectors (x_k) in R^p features space. (1)${\displaystyle J_{UPFC}\left(\mathit{X};\mathit{U},\mathit{V}\right)=\sum _{k=1}^n\sum _{i=1}^c\left(a{u}_{ik}^m{+}_{ik}^{\eta }\right)d^2\left({\mathit{x}}_k,{\mathit{v}}_i\right)+\frac{\beta }{n^2\sqrt{c}}\sum _{k=1}^n\sum _{i=1}^c\left(t_{ik}^{\eta }log{t}_{ik}^{\eta }-t_{ik}^{\eta }\right)}$

In Eq. (1), u_ik and t_ik are the fuzzy and possibilistic membership degree of k^th feature vector (x_k) to the i^th cluster respectively. The UPFC objective function has the constraints which are listed in Eq. (2). (2)${\displaystyle \sum _{i=1}^c{u}_{ik}=1;\forall k;0\le u_{ik}\le 1;a>0;b>0;m>1;\eta >1.}$

In the possibilistic algorithms, the parameter m is a fuzziness exponent and the parameter η is a typicality exponent. Both these parameters are usually set to 2. In the objective function in Eq. (1), the parameters a and b are weighing coefficients for setting the relative importance of fuzziness and typicality respectively. Generally, both of these coefficients are equally set to 1.

To minimize J_UPFC, the typicality degrees t_ik and the membership degrees u_ik are recalculated in each iteration step with Eqs. (3) and (4) respectively. (3)${\displaystyle u_{ik}={\left(\sum _{j=1}^c{\left(\frac{d\left({\mathit{x}}_k,{\mathit{v}}_i\right)}{d\left({\mathit{x}}_k,{\mathit{v}}_j\right)}\right)}^{2/\left(m-1\right)}\right)}^{-1}\forall i,k}$ (4)${\displaystyle t_{ik}=exp\left(\frac{bn\sqrt{c}{d}^2\left({\mathit{x}}_k,{\mathit{v}}_i\right)}{\beta }\right)\forall i.}$

As seen from the formula in Eq. (4), a typicality degree is a possibilistic measure indicating the membership degree of a data point to clusters. The value β is is distance variance, and calculated as seen in Eq. (5). (5)${\displaystyle \beta =\frac{1}{n}\sum _{k=1}^n{d}^2\left({\mathit{x}}_k,\overline{x}\right);\overline{x}=\frac{1}{n}\sum _{k=1}^n{\mathit{x}}_k}$

The formula in Eq. (6) is used to update the cluster centers in each iteration. The UPFC algorithm stops when a predefined convergence level is achieved. (6)${\displaystyle v_i=\frac{\sum _{k=1}^n\left(a{u}_{ik}^m{+}_{ik}^{\eta }\right){\mathit{x}}_k}{\sum _{k=1}^n\left(a{u}_{ik}^m{+}_{ik}^{\eta }\right)}\forall i}$

Approach 1

While the fuzzy clustering algorithms determine the fuzzy membership degree of fuzzy data points to any cluster, they do not evaluate typicality according to their distances from cluster centers. For example, suppose there are two data points A and B, both they have 50% fuzzy membership degree to two different clusters. If their distances from the cluster centers are, say, 2r and 5r, the latter will be a more atypical data point as it is further away from the cluster centers. When compared to fuzzy membership degrees, typicality degrees from the fuzzy and possibilistic clustering algorithms reveal the distinction between the highly atypical and the less atypical members of the clusters. This means that all fuzzy points for a cluster are not equivalent: some are more typical and some are not. This implies that typicality is distinct from a simple similarity to the cluster center because it also involves a dissimilarity notion. Therefore, in our proposed algorithms we used the typicality results from UPFC algorithm.

In our first approach, a data point which is not a member of any cluster is evaluated as an outlier. In such case, its average typicality to all clusters should be less than a predefined possibilistic threshold level (α). This approach checks whether the average typicality of kth feature vector (x_k) to all clusters exceeds a threshold level, which is a user-defined possibility degree for evaluating a data point as an outlier.

The test function in Eq. (7) is used to label a data point as an outlying data point. If the average typicality of x_k to all clusters (c clusters) is less than α, it is considered as a highly atypical data point and flagged as an outlier in the data set (see in Algorithm 1).

Choosing an appropriate α value is crucial in this approach. According to our tests on several experimental data sets, we determined that a threshold level of 1% (α = 0.01) may be sufficient to find the outliers on the results of UPFC runs, and therefore we recommend to use it as the default value for approach on large data sets. (7)${\displaystyle is.outlier\left(x_k\right)=\left\{\begin{array}{c}1if\left(\sum _{i=1}^c{t}_{ik}/c\right)\le \alpha \hfill \\ 0otherwise\hfill \end{array}\right.}$

Approach 2

In the second approach, it is assumed that a data point is an outlier if it is atypical for every cluster in ad a data set. As seen in Eq. (8), if the typicality of the feature vector x_k for all clusters is less than a user-defined threshold level (α), it is evaluated as an outlier, otherwise, it is normal (see in Algorithm 1). As expected with this function, larger threshold values may lead to less number of outliers whereas smaller ones may lead to much more number of outliers. However, we recommend to use a default threshold typicality level of 5% (α = 0.05) for finding the most probable outliers in data sets, one can increase it to higher levels to make the detection task looser. But, in this case, the probability of treating normal data points as outliers (false positives) also increase. In our experiments, a threshold level up to 10% was sufficient to detect the simulated outliers in many runs of UPFC on several synthetic data sets. One can look for an appropriate threshold with trial and error approach for different data sets or can use a prior known level of threshold for a specific application domain. (8)${\displaystyle is.outlier\left({\mathit{x}}_k\right)=\left\{\begin{array}{c}1if{t}_{ik}\le \alpha ;\forall i\hfill \\ 0otherwise\hfill \end{array}\right.}$

In some cases, even so, the proposed approaches can find most of the outliers in a data set, collective outliers do exist in it, and their presence should be checked. These are the outlying data points in small or sparse clusters. Thus, all members of small clusters could be seen as the collective outliers according to the third assumption is given in Chandola, Banerjee & Kumar (2009). If the number of members of a cluster is less than a threshold cluster size it can be flagged as a small cluster. Several formulas have been presented to calculate a threshold size for determining small clusters. For example, Loureiro, Torgo & Soares (2004) stated that a cluster is small if its size less than half of the average size of c clusters (n/2c). Santos-Pereira & Pires (2002) proposed that clusters whose size of 2p + 2 can be considered as small outlier clusters. However, the mentioned proposals might be useful for small data sets they may not work well for larger and high dimensional data sets. For computing a more suitable threshold size to determine small clusters (tcs), we propose to use the formula in Eq. (9). This is a simple formula based on the expected cluster size, which is weighted with the logarithm of the number of features in an examined data set. (9)${\displaystyle tcs=\frac{log2n}{c}log2p}$