A genetic integrated fuzzy classifier
Introduction
Clustering and classification problems can be often hard or vague, in spite of the simplicity of their formulation. For example, a good knowledge of the probability distribution of the feature space is not always available; the Gaussian assumption, that is often considered valid, hardly applies; linear separation between classes is usually a rough approximation.
Moreover, features and parameters, representing each element of the universe, U, may have a non-numerical nature, so that, their interpretation becomes subjective and does not follow any statistical model. In such a situation, data-analysis system should include a suitable “fuzzy” representation of both data-model and expert-knowledge.
Fuzzy-clustering (Bezdek, 1987, Ruspini, 1969, Backer and Jain, 1981) has been introduced to tackle these situations. Fuzzy-clustering algorithms are, usually, objective functionals; i.e. the clustering criteria is based on the maximisation (minimisation) of a gain (loss) function, that is computed, for instance, from a possibility distribution (Yager, 1980) of the data that is provided by experts. The fuzzy c-mean (FCM) algorithm provides fairly good results when applied to the analysis of multi-spectra thematic maps (Cannon et al., 1986) and to the analysis of Magnetic Resonance Images (Di Gesù et al., 1991).
Moreover, the starting of multi-sensors experiments in the astronomy, high energy physics, and remote sensing domains involves the fusion of heterogeneous data and the combination of several methods embedded in the so called integrated data analysis systems (Gerianotis and Chau, 1990). The main motivation for considering combined technique is that even humans take decision models by using more than one evaluation paradigm, and usually complex decisions are taken by considering the evaluation of more than one expert.
In this paper we introduce two Integrated Fuzzy Classification (IFC) schemes. They are grounded on the weighted combination of classifier Cls with 1 < s ⩽ S. The accuracy and the precision of each classifier can be represented by a weight, π, defined in the interval [0, 1]. In (Di Gesù, 1994) it is shown the application of an integration method for the segmentation of medical MRI images. The explicit form of the IFC depends on the modality of the integration (sequential, parallel, and hybrid). In the following, we will consider the parallel and sequential integration schemes.
In the last decade, several methods for the integration of multiple classifiers have been developed (Dietterich, 2000, Valentini and Masulli, 2002). Recently, the fusion of classifiers has been performed by weighting the training samples, where a training sample with a high weight has a larger probability of being used in the training of the next classifier. To this purpose a sequential integration modality is used. Adaptive Boosting (Freund and Schapire, 1995, Shapire et al., 1998) is an example of such integration algorithm. Other approaches assign different feature subsets to each single classifier (Kuncheva and Jain, 2000), integrate individual decision boundaries (e.g. mixture of experts (Jordan and Jacobs, 1994), ensemble averaging (Hashem, 1997)).
Moreover, the search for the best combination, corresponding to the near-optimal classes separation, can be always formulated as an optimization problem. This suggests the use of a genetic approach (Holland, 1975, Goldberg, 1989, Michalewicz, 1996) to face up to IFC. Genetic algorithms (GA) differ from more traditional optimization techniques in that they involve a search starting from a population of solutions, and not from a single point. Each iteration of a GA involves a competitive selection that weeds out poor solutions. Solutions with a higher fitness are recombined with other solutions by swapping parts of a solution with another. Solutions are also “mutated” by making a small change to a single element of it. This generation mechanism provides the ability to span the solution space escaping very often (but not always) from local maxima (minima). GA have been already applied successfully to clustering problems (Hall and Ozyurt, 1999, Lo Bosco, 2001).
An example of classifier integration has been proposed in (Mitra et al., 2001, Pal et al., 2003), where, a modular evolutionary strategy is presented to design a hybrid connectionist system that is based on the optimal combination of classification subtasks performed by a set of Multiple Layer Perceptron.
In the following the IFC controlled by a genetic procedure are named Genetic-IFC. Two examples of Genetic-IFC algorithm are described in detail. They are supervised and both of them integrate three classifiers that run a nearest neighbor (NN) algorithm using three different distance functions. The first algorithm performs the classification by a direct assignment of each element to one of the classes, we name it all-against-all (AAA). The second one performs a tree classification of one class respect the remaining classes. The procedure is repeated for the total number of classes and we name it one-against-remaining (OAR).
Moreover, the genetic optimization yields the Genetic-IFC entirely free from the tuning of “ad hoc” parameters, and this is a very useful property for a classifier.
Both versions of Genetic-IFC have been validated and tested on three data-sets. The first represents a breast cancer databases from the University of Wisconsin, the second is a waveform data-set. Both of them have been retrieved from the public domain (ftp://ftp.ics.uci.edu/pub/machine-learning-databases).
The third data-set represents four types of biological cells (classes): bacteria, white blood cell casts, non squamous renal epithelial, and non squamous transitional epithelial named A, B, C, and D respectively (see Fig. 1). From the figure it is evident that the classification is quite hard because non squamous renal epithelial and non squamous transitional epithelial classes have similar shapes, features and texture. This data-set has been kindly provided by IRIS Diagnostic, CA, USA.
The same data set has been classified using a feed-forward neural network (Bishop, 1996) and a Support Vector Machine (SVM) (Cristianini and Shawe-Taylor, 2000, Guyon et al., 1993) for comparison purposes. A naive Bayesian classifier has been also considered for comparison.
In Section 2 the integration problem is outlined. Section 3 describes the all-against-all and one-against-remaining classifiers and the genetic optimisation procedure. Section 4 describes an application of the proposed methods to the classification of biological cells. Experimental results are shown in Section 5. Section 6 is dedicated to final remarks and future developments.
Section snippets
Integrated fuzzy classification
A classification problem can be stated as follows: given a universe U ⊂ R+ find the partition of K elements (clusters) ℘(U) ≡ {C1,C2,…,Ck,…,CK} such that ∀x ∈ U, x ∈ Ck ⇔ k = argmin1⩽i⩽K(δi(x)); where δ(k) is said discriminant function. In the following, C ≡ {Cl1,Cl2, …, ClS} is the sets of S classifiers, M ≡ {M(1), …, M(s), …, M(S)} are K × K confusion matrices derived from each classifiers, and ΠM ≡ {M(1) /π1, M(2)/π2, …, M(s)/πs, …, M(S)/πS} is the initial possibility distribution of M, with πs ∈ [0,1]. The meaning of ΠM
The Genetic-IFC
In the following two Genetic-IFC are described. The first one is based on IFC-parallel strategy, the latter is an example of IFC-sequential.
All-against-all (Fig. 2). Each classifier provides its own confusion matrix and possibility distribution, then the genetic optimisation (GO) procedure is applied as described later.
One-against-remaining (Fig. 3). Here, the all-against-all procedure is applied K − 1 times for two classes. In fact, at the kth stage the kth class is discriminated against the
An application to cell classification
In the following we describe an application of Genetic-IFC to the classification of biological cells.
Experimental results
In the following we describe the experiments performed on three different data-sets:
- •
Breast cancer databases from the University of Wisconsin (ftp://ftp.ics.uci.edu/pub/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.data);
- •
Waveform (ftp://ftp.ics.uci.edu/pub/machine-learning-databases/waveform/waveform.data.Z, Breiman et al., 1984);
- •
Urine analysis cells.
The two classifiers, (AAA and OAR), have been compared with a two layer feed-forward neural net (FFNN) and a Support
Conclusions
This paper introduces a new paradigm for classifiers integration that is based on a GA optimization approach, searching for the best integration parameters. The Genetic-IFC has been applied to the classification of biological cells by integrating three different classifiers. The genetic paradigm has been chosen because it allows us to search in a large solution space that can be defined subjectively. In our case the performance of three separated classifiers are combined to reach a better
References (34)
Integrated Fuzzy Clustering
Fuzzy Sets and Systems
(1994)- et al.
Evolutionary modular design of rough knowledge-based network using fuzzy attributes
Neurocomputing
(2001) - et al.
A clustering performance measure based on fuzzy set decomposition
IEEE Trans PAMI
(1981) Pattern Recognition with Fuzzy Objective Function Algorithms
(1987)Neural Networks for Pattern Recognition
(1996)- et al.
Classification and regression trees
Wadsworth Int. Group
(1984) - et al.
Segmentation of a thematic mapper image using the fuzzy c-means clustering algorithm
IEEE Trans. Geosci. Remote Sensing
(1986) - et al.
An Introduction to Support Vector Machines and Other Kernel-based Learning Methods
(2000) - et al.
Clustering algorithms for MRI
- et al.
Classification based on iterative object symmetry transform
Proc. IEEE Int. Conf. Image Anal. Process
(2003)
Symmetry in computer vision
Ensemble methods in machine learning
Robust data fusion for multisensor detection systems
IEEE Trans. Pattern Informat. Theory
Cited by (7)
Market segmentation and ideal point identification for new product design using fuzzy data compression and fuzzy clustering methods
2012, Applied Soft Computing JournalColony image acquisition and genetic segmentation algorithm and colony analyses
2012, Proceedings of SPIE - The International Society for Optical EngineeringEvolutionary computer vision
2007, Proceedings of GECCO 2007: Genetic and Evolutionary Computation Conference, Companion MaterialData analysis and bioinformatics
2007, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)Combining one class fuzzy KNN's
2007, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)A fast genetic fuzzy approach and its application in classifier fusion
2006, International Conference on Signal Processing Proceedings, ICSP