A genetic integrated fuzzy classifier

https://doi.org/10.1016/j.patrec.2004.08.004Get rights and content

Abstract

This paper introduces a new classifier, that is based on fuzzy-integration schemes controlled by a genetic optimisation procedure. Two different types of integration are proposed here, and are validated by experiments on real data sets of biological cells. The performance of our classifier is tested against a feed-forward neural network and a Support Vector Machine. Results show the good performance and robustness of the integrated classifier strategies.

Introduction

Clustering and classification problems can be often hard or vague, in spite of the simplicity of their formulation. For example, a good knowledge of the probability distribution of the feature space is not always available; the Gaussian assumption, that is often considered valid, hardly applies; linear separation between classes is usually a rough approximation.

Moreover, features and parameters, representing each element of the universe, U, may have a non-numerical nature, so that, their interpretation becomes subjective and does not follow any statistical model. In such a situation, data-analysis system should include a suitable “fuzzy” representation of both data-model and expert-knowledge.

Fuzzy-clustering (Bezdek, 1987, Ruspini, 1969, Backer and Jain, 1981) has been introduced to tackle these situations. Fuzzy-clustering algorithms are, usually, objective functionals; i.e. the clustering criteria is based on the maximisation (minimisation) of a gain (loss) function, that is computed, for instance, from a possibility distribution (Yager, 1980) of the data that is provided by experts. The fuzzy c-mean (FCM) algorithm provides fairly good results when applied to the analysis of multi-spectra thematic maps (Cannon et al., 1986) and to the analysis of Magnetic Resonance Images (Di Gesù et al., 1991).

Moreover, the starting of multi-sensors experiments in the astronomy, high energy physics, and remote sensing domains involves the fusion of heterogeneous data and the combination of several methods embedded in the so called integrated data analysis systems (Gerianotis and Chau, 1990). The main motivation for considering combined technique is that even humans take decision models by using more than one evaluation paradigm, and usually complex decisions are taken by considering the evaluation of more than one expert.

In this paper we introduce two Integrated Fuzzy Classification (IFC) schemes. They are grounded on the weighted combination of classifier Cls with 1 < s  S. The accuracy and the precision of each classifier can be represented by a weight, π, defined in the interval [0, 1]. In (Di Gesù, 1994) it is shown the application of an integration method for the segmentation of medical MRI images. The explicit form of the IFC depends on the modality of the integration (sequential, parallel, and hybrid). In the following, we will consider the parallel and sequential integration schemes.

In the last decade, several methods for the integration of multiple classifiers have been developed (Dietterich, 2000, Valentini and Masulli, 2002). Recently, the fusion of classifiers has been performed by weighting the training samples, where a training sample with a high weight has a larger probability of being used in the training of the next classifier. To this purpose a sequential integration modality is used. Adaptive Boosting (Freund and Schapire, 1995, Shapire et al., 1998) is an example of such integration algorithm. Other approaches assign different feature subsets to each single classifier (Kuncheva and Jain, 2000), integrate individual decision boundaries (e.g. mixture of experts (Jordan and Jacobs, 1994), ensemble averaging (Hashem, 1997)).

Moreover, the search for the best combination, corresponding to the near-optimal classes separation, can be always formulated as an optimization problem. This suggests the use of a genetic approach (Holland, 1975, Goldberg, 1989, Michalewicz, 1996) to face up to IFC. Genetic algorithms (GA) differ from more traditional optimization techniques in that they involve a search starting from a population of solutions, and not from a single point. Each iteration of a GA involves a competitive selection that weeds out poor solutions. Solutions with a higher fitness are recombined with other solutions by swapping parts of a solution with another. Solutions are also “mutated” by making a small change to a single element of it. This generation mechanism provides the ability to span the solution space escaping very often (but not always) from local maxima (minima). GA have been already applied successfully to clustering problems (Hall and Ozyurt, 1999, Lo Bosco, 2001).

An example of classifier integration has been proposed in (Mitra et al., 2001, Pal et al., 2003), where, a modular evolutionary strategy is presented to design a hybrid connectionist system that is based on the optimal combination of classification subtasks performed by a set of Multiple Layer Perceptron.

In the following the IFC controlled by a genetic procedure are named Genetic-IFC. Two examples of Genetic-IFC algorithm are described in detail. They are supervised and both of them integrate three classifiers that run a nearest neighbor (NN) algorithm using three different distance functions. The first algorithm performs the classification by a direct assignment of each element to one of the classes, we name it all-against-all (AAA). The second one performs a tree classification of one class respect the remaining classes. The procedure is repeated for the total number of classes and we name it one-against-remaining (OAR).

Moreover, the genetic optimization yields the Genetic-IFC entirely free from the tuning of “ad hoc” parameters, and this is a very useful property for a classifier.

Both versions of Genetic-IFC have been validated and tested on three data-sets. The first represents a breast cancer databases from the University of Wisconsin, the second is a waveform data-set. Both of them have been retrieved from the public domain (ftp://ftp.ics.uci.edu/pub/machine-learning-databases).

The third data-set represents four types of biological cells (classes): bacteria, white blood cell casts, non squamous renal epithelial, and non squamous transitional epithelial named A, B, C, and D respectively (see Fig. 1). From the figure it is evident that the classification is quite hard because non squamous renal epithelial and non squamous transitional epithelial classes have similar shapes, features and texture. This data-set has been kindly provided by IRIS Diagnostic, CA, USA.

The same data set has been classified using a feed-forward neural network (Bishop, 1996) and a Support Vector Machine (SVM) (Cristianini and Shawe-Taylor, 2000, Guyon et al., 1993) for comparison purposes. A naive Bayesian classifier has been also considered for comparison.

In Section 2 the integration problem is outlined. Section 3 describes the all-against-all and one-against-remaining classifiers and the genetic optimisation procedure. Section 4 describes an application of the proposed methods to the classification of biological cells. Experimental results are shown in Section 5. Section 6 is dedicated to final remarks and future developments.

Section snippets

Integrated fuzzy classification

A classification problem can be stated as follows: given a universe U  R+ find the partition of K elements (clusters) ℘(U)  {C1,C2,…,Ck,…,CK} such thatx  U, x  Ck  k = argmin1⩽iK(δi(x)); where δ(k) is said discriminant function. In the following, C  {Cl1,Cl2, …, ClS} is the sets of S classifiers, M  {M(1), …, M(s), …, M(S)} are K × K confusion matrices derived from each classifiers, and ΠM  {M(1) /π1, M(2)/π2, …, M(s)/πs, …, M(S)/πS} is the initial possibility distribution of M, with πs  [0,1]. The meaning of ΠM

The Genetic-IFC

In the following two Genetic-IFC are described. The first one is based on IFC-parallel strategy, the latter is an example of IFC-sequential.

All-against-all (Fig. 2). Each classifier provides its own confusion matrix and possibility distribution, then the genetic optimisation (GO) procedure is applied as described later.

One-against-remaining (Fig. 3). Here, the all-against-all procedure is applied K  1 times for two classes. In fact, at the kth stage the kth class is discriminated against the

An application to cell classification

In the following we describe an application of Genetic-IFC to the classification of biological cells.

Experimental results

In the following we describe the experiments performed on three different data-sets:

  • Breast cancer databases from the University of Wisconsin (ftp://ftp.ics.uci.edu/pub/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.data);

  • Waveform (ftp://ftp.ics.uci.edu/pub/machine-learning-databases/waveform/waveform.data.Z, Breiman et al., 1984);

  • Urine analysis cells.


The two classifiers, (AAA and OAR), have been compared with a two layer feed-forward neural net (FFNN) and a Support

Conclusions

This paper introduces a new paradigm for classifiers integration that is based on a GA optimization approach, searching for the best integration parameters. The Genetic-IFC has been applied to the classification of biological cells by integrating three different classifiers. The genetic paradigm has been chosen because it allows us to search in a large solution space that can be defined subjectively. In our case the performance of three separated classifiers are combined to reach a better

References (34)

  • V. Di Gesù

    Integrated Fuzzy Clustering

    Fuzzy Sets and Systems

    (1994)
  • S. Mitra et al.

    Evolutionary modular design of rough knowledge-based network using fuzzy attributes

    Neurocomputing

    (2001)
  • E. Backer et al.

    A clustering performance measure based on fuzzy set decomposition

    IEEE Trans PAMI

    (1981)
  • J.C. Bezdek

    Pattern Recognition with Fuzzy Objective Function Algorithms

    (1987)
  • C.M. Bishop

    Neural Networks for Pattern Recognition

    (1996)
  • L. Breiman et al.

    Classification and regression trees

    Wadsworth Int. Group

    (1984)
  • R.L. Cannon et al.

    Segmentation of a thematic mapper image using the fuzzy c-means clustering algorithm

    IEEE Trans. Geosci. Remote Sensing

    (1986)
  • N. Cristianini et al.

    An Introduction to Support Vector Machines and Other Kernel-based Learning Methods

    (2000)
  • V. Di Gesù et al.

    Clustering algorithms for MRI

  • V. Di Gesù et al.

    Classification based on iterative object symmetry transform

    Proc. IEEE Int. Conf. Image Anal. Process

    (2003)
  • V. Di Gesù

    Symmetry in computer vision

  • T.G. Dietterich

    Ensemble methods in machine learning

  • Freund, Y., Schapire, R.E., 1995. A decision-theoretic generalization of on-line learning and an application to...
  • ...
  • ...
  • ...
  • E. Gerianotis et al.

    Robust data fusion for multisensor detection systems

    IEEE Trans. Pattern Informat. Theory

    (1990)
  • Cited by (7)

    • Colony image acquisition and genetic segmentation algorithm and colony analyses

      2012, Proceedings of SPIE - The International Society for Optical Engineering
    • Evolutionary computer vision

      2007, Proceedings of GECCO 2007: Genetic and Evolutionary Computation Conference, Companion Material
    • Data analysis and bioinformatics

      2007, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    • Combining one class fuzzy KNN's

      2007, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    • A fast genetic fuzzy approach and its application in classifier fusion

      2006, International Conference on Signal Processing Proceedings, ICSP
    View all citing articles on Scopus
    View full text