Abstract
Recent classifier combination frameworks have proposed several ways of weakening a learning set and have shown that these weakening methods improve prediction accuracy. In the present paper we focus on learning set sampling (Breiman’s bagging) and random feature subset selections (Bay’s Multiple Feature Subsets). We present a combination scheme labeled ‘Bagfs’, in which new learning sets are generated on the basis of both bootstrap replicates and selected feature subsets. The performances of the three methods (Bagging, MFS and Bagfs) are assessed by means of a decision-tree inducer (C4.5) and a majority voting rule. In addition, we also study whether the way in which weak classifiers are created has a significant influence on the performance of their combination. To answer this question, we undertook the strict application of the Cochran Q test. This test enabled us to compare the three weakening methods together on a given database, and to conclude whether or not these methods differ significantly. We also used the McNemar test to compare algorithms pair by pair. The first results, obtained on 14 conventional databases, show that on average, Bagfs exhibits the best agreement between prediction and supervision. The Cochran Q test indicated that the weak classifiers so created significantly influenced combination performance in the case of at least 4 of the 14 databases analyzed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ali and Pazzani. Error reduction through learning multiple descriptions. Machine Learning, 24:173–202, 1996.
R. Battiti and A.M. Colla. Democracy in neural nets: voting schemes for classification. Neural Networks, 7(4):691–708, 1995.
Stephen D. Bay. Nearest neighbor classification from multiple feature subsets. In Proceedings of the International Conference on Machine Learning, Madison, Wisc., 1998. Morgan Kaufmann Publishers.
C. Blake, E. Keogh, and C.J. Merz. Uci repository of machine learning databases. http://www.ics.uci.edu/ mlearn/MLRepository.html. Irvine,CA: University of California,Department of Information and Computer Science, 1998.
Leo Breiman. Bagging predictors. Machine Learning, 24, 1996.
Leo Breiman. Bias, variance and arcing classifiers. Technical report, Statistics department. University of California, 1996.
Thomas G. Dietterich. An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting and randomization. Machine Learning, pages 1–22, 1999.
Ioav Freund and Robert E. Schapire. Experiments with a new boosting algorithm. In In Prod. of the 13th International Conference on Machine Learning, pages 146–148. Morgan Kaufmann, 1996.
T.K Ho, Jonathan J. Hull, and Sargur N. Srihari. Decision combination in multiple classifier systems. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(1):66–75, 1994.
Y. S. Huang and Ching Y. Suen. A method of combining multiple experts for the recognition of unconstrained handwritten numerals. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(1), 1995.
Joseph Kittler. Combining classifiers: a theoretical framework. Pattern Analysis and Applic., 1:18–27, 1998.
Ron Kohavi and Clayton Kunz. Option decision trees with majority votes. In Proceedings of the Fourtheeth International Conference on Machine Learning, pages 161–169, San Francisco, CA, 1997. Morgan Kaufmann.
Louisa Lam and Ching Y. Suen. Application of majority voting to pattern recognition: an analysis of its behavior and performance. IEEE tr. on systems, man and cybernetics, 27(5), 1997.
J.R. Quinlan. C4.5: Programs For Machine Learning. Morgan Kaufmann Publishers, San Mateo,California, 1993.
J.R. Quinlan. Bagging, boosting, and c4.5. In Proceedings of the Thirteenth National Conference on Artificial Intelligence, pages 725–730, 1996.
G.H. Rosenfield and K. Fitzpatrick-Lins. A coefficient of agreement as a measure of thematic classification accuracy. Photogrammetric Engineering and Remote Sensing. 52(2):223–227, February 1986.
Bernard Rosner. Fundamentals of Bio statistics. Duxbury Press (ITP), Belmont, CA, USA, 4th edition, 1995.
S. Siegel and N.J. Castellan. Nonparametric Statistics for the behavioral sciences. McGraw-Hill, second edition, 1988.
K. Tumer and J. Ghosh. Classifier combining: analytical results and implications. In Proceedings of the National Conference on Artificial Intelligence, Portland, OR, 1996.
L. Xu, A. Krzyzak, and Ching Y. Suen. Methods of combining multiple classifiers and their applications to handwriting recogntion. IEEE tr. on systems, man and cybernetics, 22(3):418–435, 1992.
Zijian Zheng. Generating classifier committees by stochastically selecting both attributes and training examples. In Proceedings of the 5th Pacific Rim International Conferences on Artificial Intelligence (PRICAI’98), pages 12–23. Barlin Springer-Verlag, 1998.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2000 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Latinne, P., Debeir, O., Decaestecker, C. (2000). Different Ways of Weakening Decision Trees and Their Impact on Classification Accuracy of DT Combination. In: Multiple Classifier Systems. MCS 2000. Lecture Notes in Computer Science, vol 1857. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45014-9_19
Download citation
DOI: https://doi.org/10.1007/3-540-45014-9_19
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-67704-8
Online ISBN: 978-3-540-45014-6
eBook Packages: Springer Book Archive