Electrical fuzzy C-means: A new heuristic fuzzy clustering algorithm

: Many heuristic and meta-heuristic algorithms have been successfully applied in the literature to solve the clustering problems. The algorithms have been created for partitioning and classifying a set of data because of two main purposes: at first, for the most compact clusters, second, for the maximum separation be-tween clusters. In this paper, we propose a new heuristic fuzzy clustering algorithm based on electrical rules. The laws of attraction and repulsion of electric charges in an electric field are conducted the same as the target of clustering. The electrical fuzzy C-means (EFCM) algorithm proposed in this article use the electrical rules in electric fields and Coulomb’s law to obtain the better and the realest partitioning, having respect to the maximum separation of clusters and the maximum compactness within clusters. Computational results show that our proposed algorithm in comparison with fuzzy C-means (FCM) algorithm as a well-known fuzzy clustering algorithm have good performance.


PUBLIC INTEREST STATEMENT
Clustering plays a key role in searching for structures in data and involves the task of dividing data points into clusters so that items in the same class are as similar as possible and items in different classes are as dissimilar as possible. In real-world cases, fuzzy clustering will be a better choice for the data and the data points can belong to more than one cluster. The membership grades of each data points represent the degree to which the point belongs to each cluster. In this paper a heuristic algorithm based on electrical rules is proposed to solve fuzzy clustering problem. The laws of attraction and repulsion of electric charges are conducted the same as the target of clustering. The algorithm obtains the realest partitioning, having respect to the maximum separation of clusters and the maximum compactness within clusters.

Introduction
Form a general point of view, pattern recognition is defined as the process of searching for data structures and the related classification into certain categories, in which the association among the intracategorical and inter-categorical structures is high and low, respectively. Clustering is the most fundamental and significant issue in pattern recognition and is defined as a form of data compression, in which a large number of samples are converted into a small number of representative prototypes or clusters (Klir & Yuan, 2003). Clustering plays a key role in searching for structures in data and involves the task of dividing data points into homogeneous classes or clusters so that items in the same class are as similar as possible and items in different classes are as dissimilar as possible (Mehdizadeh, Sadi-Nezhad, & Tavakkoli-Moghaddam, 2008). It is a method creating groups of objects so that objects within one cluster are similar and objects in different clusters are dissimilar. In the last few years clustering has played a critical role in different domain of science and engineering applications such as image processing (Xia, Feng, Wang, Zhao, & Zhang, 2007;Yang, Wu, Wang, & Jiao, 2010), anomaly detection (Friedman, Last, Makover, & Kandel, 2007;Moshtaghi et al., 2011), medicine (Liao, Lin, & Li, 2008), construction management (Cheng & Leu, 2009), marketing (Kim & Ahn, 2008), data retrieval (Abraham, Das, & Konar, 2006;Mahdavi, Haghir Chehreghani, Abolhassani, & Forsati, 2008;Gil-García & Pons-Porrata, 2010), reliability (Taboada & Coit, 2007), portfolio optimization (Chen & Huang, 2009), cell formation problem (Mehdizadeh & Tavakkoli-Moghaddam, 2009;Mehdizadeh, 2009) selecting supplier (Che, 2012;Mehdizadeh & Tavakkoli-Moghaddam, 2007), supplier clustering (Mehdizadeh, 2009) and data envelopment analysis (Po, Guh, & Yang, 2009;Ben-Arieh & Gullipalli, 2012), support vector machines (Sabzekar & Naghibzadeh, 2013).
In real-world cases, there are very often no sharp boundaries between clusters so that fuzzy clustering will be a better choice for the data. Membership degrees between zero and one are used in fuzzy clustering instead of crisp assignments of data to clusters. In non-fuzzy (crisp environment) or hard clustering, data are divided into crisp clusters, whose data point belongs to exactly one cluster. In fuzzy clustering, these data points can belong to more than one cluster, under these circumstances, the membership grades of each of the data points represent the degree to which the point belong to each cluster (Mehdizadeh & Tavakkoli-Moghaddam, 2009).
In literature, many algorithms such as heuristic and meta-heuristic have been proposed for solving fuzzy clustering problems. One of the most applicable methods of fuzzy clustering is fuzzy C-means (FCM) algorithm. FCM is an efficient tool used for fuzzy clustering problems. This method has been successfully adapted to solve the fuzzy clustering problem. However, this problem is a combinatorial optimization problem (Zimmermann, 1996) and if the data-sets are very high dimensional or contain severe noise points, the FCM often fails to find the global optimum. In these cases, the probability of finding the global optimum can be increased by improving FCM with inspiration by natural rules. In this paper, to skip the local optimum, the FCMs algorithm is combined with the electrical rules and a new algorithm called Electrical EFCM algorithm is presented for solving fuzzy clustering problem.
In this article, the proposed fuzzy clustering algorithm uses the electrical rules and electric potential energy lies in electric fields and Coulomb's law to obtain the realest partitioning, having respect to the maximum separation of clusters and the maximum compactness within clusters. Clustering algorithms have been created for partitioning and classify a set of data because of two main purposes: first, for the most compact clusters, second, for the maximum separation between clusters. The laws of attraction and repulsion of electric charges in an electric field are conducted exactly the same as the target of clustering. Thus the charge with a negative charge at the center of a cluster of positively charged clusters are absorbed and the positively charged cluster of data centers and other negatively charged clusters, there is gravity. Clusters that formed by clustering algorithms act like the electrical loads. Different clusters have a repulsive force or a good separation between the data and they are trying to provide better compact within clusters. The proposed algorithm (EFCM) starts in a way that randomly chose the initial centers and ends by computing a unique objective function. In each iteration of this algorithm, the existing data is displayed at the beginning of the computation center of each iteration.
The remaining of this paper is organized as follows: Section 2 presents the literature review. An overview of the FCM algorithm, Coulomb's low, and electric potential energy is discussed in Section 3. In Section 4 we introduce our proposed EFCMs algorithm. Experimental results are summarized in Section 5. Finally, discussion and conclusions are presented in Section 6.

Literature review
Clustering has a long history, dating back to Aristotle (Blacel, Hansen, & Mladenovic, 2002). Clustering algorithms allocated each object to a cluster that this is the most popular problems for crisp clustering. Fuzzy logic (Zadeh, 1965) creates approximate clustering rather than crisp clustering by using fuzzy clustering problem. This problem is solved and the object can allocate to all of the clusters with a certain degree of membership (Bezdek, 1981). In literature, many algorithms such as heuristic and meta-heuristic have been proposed for solving fuzzy clustering problems. One of the most applicable methods of fuzzy clustering is FCM algorithm. The first version of the C-means algorithms was presented by Duda and Hart (1973) that known as a hard clustering algorithm. In real word, some of the data belong to multiple clusters. In order to study this problem, Dun (1974) proposed one of the first fuzzy clustering methods based on the objective function and using Euclidean distance. This algorithm was revised several times but its final version was given by Bezdek, Ehrlich and Full (1984). For solving elliptical clustering problem, Gustafson and Kesel (1979) proposed a new fuzzy clustering algorithm using covariance matrix. They used another criterion for determining the distance instead of Euclidean distance. Required normalization of the membership degrees and the sum of membership degrees being equal to one in the fuzzy clustering algorithm lead to adverse effects in clustering on the outlying and thrown away from center data. To solve this problem, the possibility clustering algorithm was proposed by Dubois and Prade (1988) and then was corrected by Krishnapuram and Keller (1993). Many reforms and improvements have been executed on this algorithm and a general algorithm for solving the problem of distinguishing various forms have been proposed by Hathaway and Bezdek (1994). The results of FCM algorithm were greatly affected by the data scattered away from the center. To solve this, a lot of algorithms have been proposed to improve the objective function (Dave, 1991;Dave & Andsen, 1997, 1998Frigui & Krishnapuram, 1996). Up to this time, for improving the Fuzzifier value on these algorithms according to the effectiveness of the Fuzzifier value, many algorithms were proposed based on FCM algorithm (Klawonn, 2004;Klawonn & Hoppner, 2003;Rousseeuw, Trauwaert, & Kaufman, 1995). Possibilistic fuzzy C-means (PCM) algorithms with optimizing the objective function leads to clusters that are not perceptible separation, clustering algorithm with modified objective function were proposed to avoid merging clusters which explain repulsion of clusters (Timm & Kruse, 2002;Timm, Borgelt, Do¨Ring, & Kruse, 2004). As the fact that a good clustering requires a good partitioning and a minimum objective function values, so many algorithm were proposed to improve PCM algorithms with respect to their possibility degree and membership degree (Pal, Pal, & Bezdek, 1997;Pal, Pal, Keller, & Bezdek, 2004). Before starting the FCM algorithms, it is necessary to use the number of clusters. Its performance strongly depends on the initial centroids' vectors and may get stuck in local optima solutions (Selim & Ismail, 1984). Generally speaking, we have never seen any difficulty and in ten to twenty-five iterations, we achieve numerical convergence. Another topic in FCM is relationship between local minima of objective function and clustering of data-set. Because of the dependence of FCM objective function to initial state, the results, usually, converge to local optimum. For declining this difficulty, on each membership matrix by FCM, it is calculated by several types of cluster validity (Bezdek, 1974). For speeding up FCM and improving drawback of FCM, researchers proposed optimization approaches for fuzzy partitioning. Some of these methods have improved FCM algorithm partitioning and some of these researches improved FCM algorithm to determine the optimal number of clusters and study on validity indexes of fuzzy clustering (Dubois & Prade, 1988;Wu, Xie, & Yu, 2003).
In recent years, researchers studied on fuzzy clustering algorithm to improve relationship between clusters in objective function with different approaches (Frigui & Krishnapuram, 1996;Keller, 2000;Klawonn & Hoppner, 2003). Some of them have used heuristic and meta-heuristic methods which inspired by social and natural rules in order to optimize the objective function of FCM. For example, Bezdek and Hathaway (1994) optimized the hard C-means (HCM) method with a genetic algorithm. Klawonn and Keller (1998) extended and applied this scheme to the FCM model. In addition, ant colony optimization (ACO) has been successfully applied to clustering problems (Handl, Knowles, & Dorigo, 2003). Similar heuristic algorithms, called ant clustering, were suggested by Kanade and Hall (2003) and (2004). Runkler (2005) introduced an ACO algorithm that explicitly minimizes the HCM and FCM cluster models. Runkler and Katz (2006) applied PSO to cluster data by considering fuzzy clustering. They introduced two new methods to minimize the two reformulated versions of the FCM objective function by PSO. A hybrid PSO and FCM clustering algorithm has been applied to clustering problem (Mehdizadeh et al., 2008). Sabzekar and Naghibzadeh (2013) used an implementation of support vector machines, namely relaxed constraints support vector machines to improve the performance of FCM algorithm. There are two hypotheses in clustering problems: (1) the most compact clusters and (2) the maximum separation between clusters. In this article, for improving the performance of FCM algorithm, we propose a new heuristic algorithm for fuzzy clustering problem based on electrical rules and called EFCMs.

EFCMs phenomenon
The EFCMs algorithm proposed in this article use the electrical potential energy in electric fields, Coulomb's law, and FCM algorithm to obtain the best and the realest partitioning, having respect to the maximum separation of clusters and the maximum compactness within clusters. In this section, a brief explanation of electrical rules applied to EFCM and FCM algorithm is presented.

FCM algorithm
FCM is one of the common algorithms of fuzzy clustering methods that proposed by Bezdek (1981) and aims to find fuzzy partitioning of a given data-set by minimizing of the basic C-means objective functional as shown in Equation (1): where c is the number of clusters; n is the number of data point; the parameter m is a real number that governs the influence of membership grades. The partition becomes fuzzier with increasing m; V i is the cluster center of cluster i; X i is the vector of data point; ‖ ‖ X k − V i ‖ ‖ 2 represents the Euclidean distance between X i and V i . The classification result can be expressed in terms of matrix U = [u ik ] c × n where u ik is the membership degree of data point k to cluster i which satisfying Equations (2)-(4): Fuzzy segmentation which used an iterative procedure is achieved with the update of membership u ik with Equation (5) and cluster centers V i by Equation (6): (1) 0 < n ∑ k=1 u ik ≤ n; i = 1, 2, … , c http://dx.doi.org/10. 1080/23311916.2016.1208397 The steps of the FCM method can be summarized in the following algorithm: Step 1: Initialize the membership matrix U with random values between 0 and 1 such that the constraints in Equations (2)-(4) are satisfied.
Step 3: Compute the objective function according to Equation (1).
Step 4: Compute a new membership matrix U by using Equation (5).
The iterations stop when the difference between the fuzzy partitions matrices in two following iterations is lower than ε.

Coulomb's law
We have been used electrical rules that described electrostatic actions between electrical points. Firstly, this rule was proposed by Charles Coulomb (Tipler, 2004). Every two charged objects will have a force on each other. Opposite charges will produce an attractive force while similar charges will produce a repulsive force. For two spherically shaped charges the formula would look like: where q 1 represents the quantity of charge on object 1 (in Coulomb's), q 2 represents the quantity of charge on object 2 (in Coulomb's), and r represents the distance of separation between the two objects (in meters). The symbol k is proportionality constant and known as the Coulomb's constant. We show how these electrical points become closer the electrical energy in Figure 1.

EFCMs algorithm
The EFCMs algorithm presented in this paper is a new fuzzy clustering algorithm based on electrical rules especially Coulomb's law. This algorithm works on the assumption that data are negative electrical charges, cluster centers are positive electrical charges, and fuzzy clusters are considered as electrical fields in an n-dimensional space. The negative electrical charges (data) in an electrical field (cluster) not only force to positive electrical charges (cluster center) in same field, but also force to positive electrical charges (cluster center) in other electrical filed (cluster) as shown in Figure 2: This algorithm is similar to FCM algorithm in minimizing the objective function. The objective function is described as follows: The first part is derived from the incoming forces from each cluster to the cluster centers (electric field) in that it shows a high density within the cluster. In this objective function we use the membership degree of objects to show the impact of the separation between clusters which has a direct impact on the objective function. This algorithm seeks to offer the most compactness of data into clusters and to satisfy the maximum separation between clusters. The algorithm used to calculate the membership degrees of data clustering and updating the membership degree matrix of a heuristic function is based on Coulomb's law. The proposed algorithm following by minimization of function J(G) is shown in Equation (8): The memberships function and center matrix update equations are as follows: where n is data vector of objects, c is number of clusters, V is cluster centers matrix, q k is the center of cluster k, q i,max u_k is the data with the highest degree of membership in a cluster k, U ik is the membership function of ith object in kth cluster, and U is membership function matrix that this membership function matrix have to satisfy Equations (3)-(5). Figure 2. The proposed clustering similar forces with the electrical forces.

Yes
We show the pseudo-code of our algorithm in Figure 3 and present the flowchart of the EFCM algorithm in Figure 4.

Experimental results
In this section, we examine performance of our proposed algorithm and compare it with FCM as a well-known fuzzy clustering algorithm on the test data-sets available in the UC Irvine Machine learning repository (Blake & Merz, 1998). We use Bezdek's validity indexes for measuring the exactness of our proposed method. All of our experiments have been implemented using MATLAB (R2011a) running on a computer with an Intel processor (Dual-core, 1.86 GHz) and 1 GB of memory.

Validity indexes
Unlike crisp clustering, fuzzy clustering allows each data point that belongs to whole clusters with a special degree of membership. It can define validity indexes for fuzzy clustering in order to seek clustering schemes in which most of the data points in the data-set exhibit a high degree of membership in one cluster. The well-known validity indexes to evaluate cluster validity which proposed by Bezdek (1974) are as follows: The partition coefficient: The partition entropy: We can see from Equation (11), the value of the PC index range is in [1/c, 1]. The closer the value of PC to 1/c, the fuzzier the clustering is. The lower value of PC is obtained when U ik = 1/c. In other words, when PC is the maximum value, the clusters are the most compact. In addition, from the definition in Equation (12), the value of the PE index range is in [0, log a c ]. The closer the value of PE to 0, the harder the clustering is. The values of PE close to the upper bound indicate that there is no clustering structure in the data-set or the algorithm is enabling to extract it. In other words when PE is the minimum value, the clusters is the most separation.

Data-sets
In order to evaluate the performance of our proposed algorithm, seven benchmark data-sets are used. The data-sets are Soybean (small), Dermatology, Breast cancer, Iris, Wine, Ecoli, and Pima, which are available in the repository of the machine learning databases (Blake & Merz, 1998). Table 1 summarizes the main characteristics of the used data-sets. The ten data-sets are described in Table 1.

Results for data-sets in terms of validity indexes
In this subsection, we display the effectiveness of our proposed algorithm which obtained based on validity indexes on data-sets described in Section 5.1. Tables 2 and 3 report the value of PC and PE as validity indexes for our proposed algorithm and FCM algorithm on mentioned data-sets in Table 1. In order to compare two algorithms, mean, best, and standard deviation of each algorithm for 50 runs computed and reported. It can be seen that EFCM algorithm has better mean, best, and standard deviation in terms of PC and PE than FCM algorithm.
Tables 4 and 5 show the comparison results of the two algorithms namely FCM and EFCM algorithms for PC and PE with different cluster number values for 50 runs. The means of the indexes for two algorithms have been reported. The best value of each data-set highlighted. Results for both indexes show that EFCM algorithm has relatively better performance for different values of cluster number.
In the Appendix A, the tables show the obtained best centroids from EFCM algorithms for the Lung cancer, Soybean (small), Dermatology, Credit approval, Breast cancer, Iris, Wine, Zoo, Ecoli, and Pima data-sets. These centroids are introduced for validating the values obtained in Tables 2 and 3. Therefore, by assigning each data-set to its center in the tables, the value of that data-set would be obtained. For example, by assigning all the objects of the Lung cancer data-set to the centroids in Tables 2 and 3, the best value for Lung cancer data-set, which is reported in Tables 2 and 3, should be obtained by EFCM algorithm. This procedure can be used for checking other data-sets.     Figures 5 and 6 show the convergence diagrams of EFCM algorithm for the best solutions for Iris and Wine data-sets. The Figures 5 and 6 show that our algorithm converges to a global optimum for Iris data-set in 28 iterations and for Wine data-set in 49 iterations.

Discussion and conclusions
Clustering is a useful technique both for data mining and data analyzing. Clustering algorithms were created for partitioning and classify a set of data because of two main purposes: the most compact clusters and the maximum separation between clusters. The laws of attraction and repulsion of electric charges in an electric field are conducted the same as the target of clustering. In this article, we have proposed a new heuristic clustering algorithm for fuzzy clustering problem based on electrical rules and Coulomb's law and called EFCMs. The proposed algorithm compared with FCM algorithm as a well-known fuzzy clustering algorithm based on two validity indexes proposed by Bezdek. The results show that the EFCM algorithm is able to achieve better solutions when compared with FCM algorithm. However, this algorithm is a heuristic algorithm and proposed to solve fuzzy