Skip to main content
SpringerLink
Log in

The use of coevolution and the artificial immune system for ensemble learning

  • Focus
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

This paper presents two new approaches for constructing an ensemble of neural networks (NN) using coevolution and the artificial immune system (AIS). These approaches are extensions of the CLONal Selection Algorithm for building ENSembles (CLONENS) algorithm. An explicit diversity promotion technique was added to CLONENS and a novel coevolutionary approach to build neural ensembles is introduced, whereby two populations representing the gates and the individual NN are coevolved. The former population is responsible for defining the ensemble size and selecting the members of the ensemble. This population is evolved using the differential evolution algorithm. The latter population supplies the best individuals for building the ensemble, which is evolved by AIS. Results show that it is possible to automatically define the ensemble size being also possible to find smaller ensembles with good generalization performance on the tested benchmark regression problems. More interestingly, the use of the diversity measure during the evolutionary process did not necessarily improve generalization. In this case, diverse ensembles may be found using only implicit diversity promotion techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Abbass HA (2002) An evolutionary artificial neural networks approach for breast cancer diagnosis. Artif Intell Med 25(3):265–281

    Article  Google Scholar 

  • Abbass HA (2003a) Pareto neuro-evolution: constructing ensemble of neural networks using multi-objective optimization. Proc IEEE Congr Evol Comput 3:2074–2080

    Google Scholar 

  • Abbass HA (2003b) Speeding up backpropagation using multiobjective evolutionary algorithms. Neural Comput 15(11):2705–2726

    Article  MATH  Google Scholar 

  • Barbosa BHG, Bui LT, Abbass HA, Aguirre LA, Braga AP (2008) Evolving an ensemble of neural networks using artificial immune systems. In: Proceedings of the 7th international conference on simulated evolution and learning. Lecture notes in computer science, Melbourne, vol 5361. Springer, Berlin, pp 121–130

  • Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140

    MathSciNet  MATH  Google Scholar 

  • Brown G, Wyatt J, Harris R, Yao X (2005) Diversity creation methods: a survey and categorisation. J Inf Fusion 6(1):5–20

    Article  Google Scholar 

  • Brown G, Wyatt J, Tino P (2005) Managing diversity in regression ensembles. J Mach Learn Res 6:1621–1650

    MathSciNet  Google Scholar 

  • Castro PD, Coelho GP, Caetano MF, Zuben FJV (2005) Designing ensembles of fuzzy classification systems: an immune approach. Lect Notes Comput Sci 3627:469–482

    Article  Google Scholar 

  • Chakraborty UK (2008) Advances in differential evolution. Springer, Berlin

    Book  MATH  Google Scholar 

  • Chandra A, Yao X (2006) Evolving hybrid ensembles of learning machines for better generalisation. Neurocomputing 69(7–9):686–700

    Article  Google Scholar 

  • Costa MA, Braga AP, Menezes BR (2003) Training neural networks with a multi-objective sliding mode control algorithm. Neurocomputing 51:467–473

    Article  Google Scholar 

  • Dasgupta D (ed) (2008) Artificial immune systems and their applications. Springer, Berlin

    Google Scholar 

  • de Castro LN, Timmis J (2002) Artificial immune systems: a new computational intelligence approach. Springer, London

    MATH  Google Scholar 

  • de Castro LN, Zuben FJV (2002) Learning and optimization using the clonal selection principle. IEEE Trans Evol Comput 6(3):239–251

    Article  Google Scholar 

  • Dietterich TG (2002) Ensemble learning. In: Arbib MA (ed) The handbook of brain theory and neural networks, 2nd edn. The MIT Press, Cambridge

    Google Scholar 

  • Drucker H, Cortes C, Jackel LD, LeCun Y, Vapnik V (1994) Boosting and other ensemble methods. Neural Comput 6(6):1289–1301

    Article  MATH  Google Scholar 

  • Friedman JH (1991) Multivariate adaptive regression splines. Ann Stat 19:1–67

    Article  MATH  Google Scholar 

  • Friedman JH (2002) Stochastic gradient boosting. Comput Stat Data Anal 38(4):367–378

    Article  MATH  Google Scholar 

  • García-Pedrajas N, Fyfe C (2007) Immune network based ensembles. Neurocomputing 70(7–9):1155–1166

    Article  Google Scholar 

  • García-Pedrajas N, Fyfe C (2008) Construction of classifier ensembles by means of artificial immune systems. J Heuristics 14(3):285–310

    Article  Google Scholar 

  • García-Pedrajas N, Ortiz-Boyer D (2007) A cooperative constructive method for neural networks for pattern recognition. Pattern Recognit 40(1):80–98

    Article  MATH  Google Scholar 

  • García-Pedrajas N, Hervßs-Martfnez C, Ortiz-Boyer D (2005) Cooperative coevolution of artificial neural network ensembles for pattern classification. IEEE Trans Evol Comput 9(3):271–302

    Article  Google Scholar 

  • Geman S, Bienenstock E, Doursat R (1992) Neural networks and the bias/variance dilemma. Neural Comput 4(1):1–58

    Article  Google Scholar 

  • Hansen LK, Salamon P (1990) Neural networks ensembles. IEEE Trans Pattern Anal Mach Intell 12(10):993–1001

    Article  Google Scholar 

  • Islam M, Yao X, Murase K (2003) A constructive algorithm for training cooperative neural network ensembles. IEEE Trans Neural Netw 14(4):820–834

    Article  Google Scholar 

  • Jin Y, Okabe T, Sendhoff B (2004) Neural network regularization and ensembling using multi-objective evolutionary algorithms. Proc IEEE Congr Evol Comput 1:1–8

    Google Scholar 

  • Krogh A, Vedelsby J (1995) Neural network ensembles, cross validation, and active learning. In: Tesauro G, Touretzky DS, Leen TK (eds) Advances in neural information processing systems, vol 7. The MIT Press, Cambridge, pp 231–238

    Google Scholar 

  • Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51(2):181–207

    Article  MATH  Google Scholar 

  • Liu Y, Yao X (1999) Ensemble learning via negative correlation. Neural Netw 12(10):1399–1404

    Article  Google Scholar 

  • Liu Y, Yao X, Higuchi T (2000) Evolutionary ensembles with negative correlation learning. IEEE Trans Evol Comput 4(4):380–387

    Article  Google Scholar 

  • McKay R, Abbass HA (2001) Anti-correlation: a diversity promotion mechanisms in ensemble learning. Aust J Intell Inf Process Syst 7(3):139–149

    Google Scholar 

  • Nguyen MH (2006) Cooperative coevolutionary mixture of experts: a neuro ensemble approach for automatic decomposition of classification problems. PhD thesis, School of Information Technology and Electrical Engineering, University of New South Wales

  • Nguyen MH, Abbass HA, McKay RI (2005) Stopping criteria for ensemble of evolutionary artificial neural networks. Appl Soft Comput 6:100–107

    Article  Google Scholar 

  • Nguyen MH, Abbass HA, Mckay RI (2006) A novel mixture of experts model based on cooperative coevolution. Neurocomputing 70:155–163

    Article  Google Scholar 

  • Opitz D, Maclin R (1999) Popular ensemble methods: an empirical study. J Artif Intell Res 11:169–198

    MATH  Google Scholar 

  • Panait L, Luke S, Wiegand RP (2006) Biasing coevolutionary search for optimal multiagent behaviors. IEEE Trans Evol Comput 10(6):629–645

    Article  Google Scholar 

  • Perrone MP, Cooper LN (1993) When networks disagree: ensemble methods for hybrid neural network. In: Mammone RJ (ed) Neural networks for speech and image processing. Chapman & Hall, London, pp 126–142

    Google Scholar 

  • Potter M (1997) The design and analysis of a computational model of cooperative coevolution. PhD thesis, George Mason University, Fairfax, Virginia

  • Potter MA, Jong KAD (2000) Cooperative coevolution: an architecture for evolving coadapted subcomponents. Evol Comput 8(1):1–29

    Article  Google Scholar 

  • Rodriguez J, Kuncheva L, Alonso C (2006) Rotation forest: a new classifier ensemble method. IEEE Trans Pattern Anal Mach Intell 28(10):1619–1630

    Article  Google Scholar 

  • Schapire RE (1990) The strength of weak learnability. Mach Learn 5(2):197–227

    Google Scholar 

  • Sollich P, Krogh A (1996) Learning with ensembles: how overfitting can be useful. In: Touretzky DS, Mozer MC, Hasselmo ME (eds) Advances in neural information processing systems, vol 8. The MIT Press, Cambridge, pp 190–196

    Google Scholar 

  • Storn R, Price K (1997) Differential evolution—a simple and efficient heuristic for global optimization. J Glob Optim 11:341–359

    Article  MathSciNet  MATH  Google Scholar 

  • Ueda N, Nakano R (1996) Generalization error of ensemble estimators. In: IEEE international conference on neural networks, Washington, DC, USA, vol 1, pp 90–95

  • Wiegand RP, Potter MA (2006) Robustness in cooperative coevolution. In: GECCO ’06: proceedings of the 8th annual conference on genetic and evolutionary computation, New York, NY, USA. ACM Press, pp 369–376

  • Witten IH, Frank E (2005) Data mining: practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco

    MATH  Google Scholar 

  • Wolpert DH (1992) Stacked generalization. Neural Netw 5(2):241–259

    Article  Google Scholar 

  • Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9):1423–1447

    Article  Google Scholar 

  • Yao X, Islam MM (2008) Evolving artificial neural network ensembles. IEEE Comput Intell Mag 3(1):31–42

    Article  Google Scholar 

  • Zhang X, Wang S, Shan T, Jiao L (2005) Selective SVMs ensemble driven by immune clonal algorithm. Lect Notes Comput Sci 3449:325–333

    Article  Google Scholar 

  • Zhou ZH, Wu J, Tang W (2002) Ensembling neural networks: many could be better than all. Artif Intell 137(1–2):239–263

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

This work has been supported by grants from CNPq, the National Council for Scientific and Technological Development, Brazil.

Author information

Authors and Affiliations

  1. Department of Electronic Engineering, Federal University of Minas Gerais, Belo Horizonte, MG, Brazil

    Bruno H. G. Barbosa, Luis A. Aguirre & Antônio P. Braga

  2. School of Information Technology and Electrical Engineering, Australian Defence Force Academy, University of New South Wales, Canberra, ACT, Australia

    Bruno H. G. Barbosa, Lam T. Bui & Hussein A. Abbass

  3. Department of Engineering, Federal University of Lavras, Lavras, MG, Brazil

    Bruno H. G. Barbosa

Authors
  1. Bruno H. G. Barbosa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lam T. Bui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hussein A. Abbass
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Luis A. Aguirre
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Antônio P. Braga
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bruno H. G. Barbosa.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Barbosa, B.H.G., Bui, L.T., Abbass, H.A. et al. The use of coevolution and the artificial immune system for ensemble learning. Soft Comput 15, 1735–1747 (2011). https://doi.org/10.1007/s00500-010-0613-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-010-0613-z

Keywords

Search

Navigation