Gonzalo Martínez-Muñoz
ABSTRACT
We present a novel ensemble pruning method based on reordering the classifiers obtained from bagging and then selecting a subset for aggregation. Ordering the classifiers generated in bagging makes it possible to build subensembles of increasing size by including first those classifiers that are expected to perform best when aggregated. Ensemble pruning is achieved by halting the aggregation process before all the classifiers generated are included into the ensemble. Pruned subensembles containing between 15% and 30% of the initial pool of classifiers, besides being smaller, improve the generalization performance of the full bagging ensemble in the classification problems investigated.
- Bakker, B., & Heskes, T. (2003). Clustering ensembles of neural network models. Neural Networks, 16, 261--269.]] Google Scholar
Digital Library
- Blake, C. L., & Merz, C. J. (1998). UCI repository of machine learning databases.]]Google Scholar
- Breiman, L. (1996a). Bagging predictors. Machine Learning, 24, 123--140.]] Google ScholarDigital Library
- Breiman, L. (1996b). Bias, variance, and arcing classifiers (Technical Report 460). Statistics Department, University of California.]]Google Scholar
- Breiman, L. (1997). Arcing the edge (Technical Report). University of California, Berkeley, CA.]]Google Scholar
- Breiman, L. (2001). Random forests. Machine Learning, 45, 5--32.]] Google ScholarDigital Library
- Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). Classification and regression trees. New York: Chapman & Hall.]]Google Scholar
- Dietterich, T. G. (2000). An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization. Machine Learning, 40, 139--157.]] Google ScholarDigital Library
- Domingos, P. (1997). Knowledge acquisition from examples via multiple models. Proc. 14th International Conference on Machine Learning (pp. 98--106). Morgan Kaufmann.]] Google ScholarDigital Library
- Esposito, R., & Saitta, L. (2004). A monte carlo analysis of ensemble classification. ICML '04: Proceedings of the twenty-first international conference on Machine learning (pp. 265--272). New York, NY, USA: ACM Press.]] Google ScholarDigital Library
- Freund, Y., & Schapire, R. E. (1995). A decision-theoretic generalization of on-line learning and an application to boosting. Proc. 2nd European Conference on Computational Learning Theory (pp. 23--37).]] Google ScholarDigital Library
- Giacinto, G., & Roli, F. (2001). An approach to the automatic design of multiple classifier systems. Pattern Recognition Letters, 22, 25--33.]] Google ScholarDigital Library
- Margineantu, D. D., & Dietterich, T. G. (1997). Pruning adaptive boosting. Proc. 14th International Conference on Machine Learning (pp. 211--218). Morgan Kaufmann.]] Google ScholarDigital Library
- Martínez-Muñoz, G., & Suáárez, A. (2004). Aggregation ordering in bagging. Proc. of the IASTED International Conference on Artificial Intelligence and Applications (pp. 258--263). Acta Press.]]Google Scholar
- Martínez-Muñoz, G., & Suáárez, A. (2005). Switching class labels to generate classification ensembles. Pattern Recognition, 38, 1483--1494.]] Google ScholarDigital Library
- Nadeau, C., & Bengio, Y. (2003). Inference for the generalization error. Machine Learning, 52, 239--281.]] Google ScholarDigital Library
- Prodromidis, A. L., & Stolfo, S. J. (2001). Cost complexity-based pruning of ensemble classifiers. Knowledge and Information Systems, 3, 449--469.]] Google ScholarDigital Library
- Schapire, R. E., Freund, Y., Bartlett, P. L., & Lee, W. S. (1998). Boosting the margin: A new explanation for the effectiveness of voting methods. The Annals of Statistics, 12, 1651--1686.]]Google Scholar
- Tamon, C., & Xiang, J. (2000). On the boosting pruning problem. Proc. 11th European Conference on Machine Learning (pp. 404--412). Springer, Berlin.]] Google ScholarDigital Library
- Webb, G. I. (2000). Multiboosting: A technique for combining boosting and wagging. Machine Learning, 40, 159--196.]] Google ScholarDigital Library
- Zhou, Z.-H., & Tang, W. (2003). Selective ensemble of decision trees. Lecture Notes in Artificial Intelligence (pp. 476--483). Berlin: Springer.]]Google ScholarCross Ref
- Zhou, Z.-H., Wu, J., & Tang, W. (2002). Ensembling neural networks: Many could be better than all. Artificial Intelligence, 137, 239--263.]] Google ScholarDigital Library
Index Terms
- Pruning in ordered bagging ensembles
Recommendations
Using boosting to prune bagging ensembles
Boosting is used to determine the order in which classifiers are aggregated in a bagging ensemble. Early stopping in the aggregation of the classifiers in the ordered bagging ensemble allows the identification of subensembles that require less memory ...
Margin distribution based bagging pruning
Bagging is a simple and effective technique for generating an ensemble of classifiers. It is found there are a lot of redundant base classifiers in the original Bagging. We design a pruning approach to bagging for improving its generalization power. The ...
An Analysis of Ensemble Pruning Techniques Based on Ordered Aggregation
Several pruning strategies that can be used to reduce the size and increase the accuracy of bagging ensembles are analyzed. These heuristics select subsets of complementary classifiers that, when combined, can perform better than the whole ensemble. The ...
Comments