Genetic algorithm-based feature set partitioning for classification problems
Section snippets
Introduction and motivation
An inducer aims to build a classifier (also known as a classification model) by learning from a set of pre-classified instances. The classifier can then be used for classifying unlabeled instances. It is well known that the required number of labeled instances for supervised learning increases as a function of dimensionality [1]. Fukunaga [2] showed that the required number of training instances for a linear classifier is linearly related to the dimensionality and for a quadratic classifier to
Related works
In this section we briefly review some of the central issues that have been addressed, and their treatment in the literature. The related work described in this section falls into three categories:
- •
First, we discuss three feature oriented tasks (namely feature selection, feature set partitioning, and feature subset-based ensemble) in pattern recognition and the relations among them.
- •
Then, we survey the usage of GAs for solving the above-mentioned tasks.
- •
The oblivious decision tree (ODT) and its
Problem formulation
In a typical classification problem, a training set of labeled examples is given. The training set can be described in a variety of languages, most frequently, as a collection of records that may contain duplicates. A vector of feature values describes each record. The notation A denotes the set of input features containing n features: and represents the class variable or the target feature. Features (sometimes referred to as attributes) are typically one of two types:
A GA method for feature set partitioning
In order to solve the problem defined in Section 3, we suggest using a GA search procedure. Fig. 4 presents the proposed process schematically. The left side in Fig. 4 specifies the creation of the ODTs ensemble based on feature set partitioning. Searching for the best partitioning is governed by a GA search. Each partitioning candidate is evaluated using a VC dimension-based evaluator. For this purpose, an ODT is generated for each feature partition. The ODT generator utilizes a caching
Experimental study
In order to illustrate the potential of the feature set partitioning approach in classification problems and to evaluate the performance of the proposed GA, a comparative experiment was conducted on benchmark datasets. The following subsections describe the experimental setup and the results obtained.
Conclusions
In this paper, we have presented a novel genetic algorithm for finding the best mutually exclusive feature set partitioning. The basic idea is to decompose the original set of features into several subsets, build a decision tree for each projection, and then combine them. This paper examines whether genetic algorithms can be useful for discovering the appropriate partitioning structure.
For this purpose we suggested a new encoding schema and fitness function that were specially designed for
Acknowledgments
The author gratefully thank the action editor and the anonymous reviewers whose constructive comments helped in improving the quality and accuracy of this paper.
About the Author—LIOR ROKACH is an assistant professor in the Department of Information System Engineering and the Program of Software Engineering of Ben-Gurion University, Israel. His research interests include artificial intelligence, pattern recognition, data mining, control of production processes and medical informatics. Dr. Rokach is the co-author of the book “Decomposition Methodology for Knowledge Discovery and Data Mining: Theory and Applications” published by World Scientific
References (58)
- et al.
Attribute bagging: improving accuracy of classifier ensembles by using random feature subsets
Pattern Recognition
(2003) Nearest neighbor classification from multiple feature subsets
Intelligent Data Anal.
(1999)- et al.
Diversity in search strategies for ensemble feature selection
Inf. Fusion
(2005) - et al.
Feature selection algorithms for the generation of multiple classifier systems
Pattern Recognition Lett.
(2004) - et al.
Comparison of algorithms that select features for pattern classifiers
Pattern Recognition
(2000) Genetic wrappers for feature selection in decision tree induction and variable ordering in Bayesian network structure learning
Inf. Sci.
(2004)Efficiently inducing determinations: a complete and systematic search algorithm that uses optimal pruning
- et al.
Induction of selective Bayesian classifiers
- et al.
Supervised classification in high-dimensional space: geometrical, statistical, and asymptotical properties of multivariate data
IEEE Trans. Syst. Man Cybern. Part C Appl. Rev.
(1998) Introduction to Statistical Pattern Recognition
(1990)
Nonparametric multivariate density estimation: a comparative study
IEEE Trans. Signal Process.
Adaptive Control Processes: A Guided Tour
Feature Extraction, Foundations and Applications, Series Studies in Fuzziness and Soft Computing
Popular ensemble methods: an empirical study
J. Artif. Res.
Neural networks and the bias/variance dilemma
Neural Comput.
Linear and order statistics combiners for pattern classification
Bagging predictors
Mach. Learn.
Experiments with a new boosting algorithm. Machine Learning
Input decimated ensembles
Pattern Anal. Appl.
The random subspace method for constructing decision forests
IEEE Trans. Pattern Anal. Mach. Intell.
Ensemble feature selection with the simple Bayesian classification in medical diagnostics
Multi-knowledge for decision making
J. Knowl. Inf. Syst.
Combining multiple K-nearest neighbor classifiers for text classification by reducts
Constructing rough decision forests
Diversity versus quality in classification ensembles based on feature selection
Using diversity in preparing ensembles of classifiers based on different feature subsets to minimize generalization error
Decomposition methodology for classification tasks—a meta decomposer framework
Pattern Anal. Appl.
Decomposition in data mining: an industrial case study
IEEE Trans. Electron. Packag. Manuf.
Cited by (0)
About the Author—LIOR ROKACH is an assistant professor in the Department of Information System Engineering and the Program of Software Engineering of Ben-Gurion University, Israel. His research interests include artificial intelligence, pattern recognition, data mining, control of production processes and medical informatics. Dr. Rokach is the co-author of the book “Decomposition Methodology for Knowledge Discovery and Data Mining: Theory and Applications” published by World Scientific Publishing and the co-editor of “The Data Mining and Knowledge Discovery Handbook” published by Springer. Dr. Rokach holds B.Sc., M.Sc. and Ph.D. in Industrial Engineering from Tel Aviv University.