Skip to main content
SpringerLink
Log in

Hybrid ensemble selection algorithm incorporating GRASP with path relinking

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The greedy randomized adaptive search procedure (GRASP) is an iterative two-phase multi-start metaheuristic procedure for a combination optimization problem, while path relinking is an intensification procedure applied to the solutions generated by GRASP. In this paper, a hybrid ensemble selection algorithm incorporating GRASP with path relinking (PRelinkGraspEnS) is proposed for credit scoring. The base learner of the proposed method is an extreme learning machine (ELM). Bootstrap aggregation (bagging) is used to produce multiple diversified ELMs, while GRASP with path relinking is the approach for ensemble selection. The advantages of the ELM are inherited by the new algorithm, including fast learning speed, good generalization performance, and easy implementation. The PRelinkGraspEnS algorithm is able to escape from local optima and realizes a multi-start search. By incorporating path relinking into GRASP and using it as the ensemble selection method for the PRelinkGraspEnS the proposed algorithm becomes a procedure with a memory and high convergence speed. Three credit datasets are used to verify the efficiency of our proposed PRelinkGraspEnS algorithm. Experimental results demonstrate that PRelinkGraspEnS achieves significantly better generalization performance than the classical directed hill climbing ensemble pruning algorithm, support vector machines, multi-layer perceptrons, and a baseline method, the best single model. The experimental results further illustrate that by decreasing the average time needed to find a good-quality subensemble for the credit scoring problem, GRASP with path relinking outperforms pure GRASP (i.e., without path relinking).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Yu L, Wang S, Lai KK (2008) Credit risk assessment with a multistage neural network ensemble learning approach. Expert Syst Appl 34:1434–1444

    Article  Google Scholar 

  2. Wang G, Hao J, Ma J, Jiang H (2011) A comparative assessment of ensemble learning for credit scoring. Expert Syst Appl 38:223–230

    Article  Google Scholar 

  3. Hand DJ, Henley WE (1997) Statistical classification methods in consumer credit scoring: a review. J R Stat Soc A Stat Soc 160:523–541

    Article  Google Scholar 

  4. Huang Z, Chen H, Hsu C-J, Chen W-H, Wu S (2004) Credit rating analysis with support vector machines and neural networks: a market comparative study. Decis Support Syst 37:543–558

    Article  Google Scholar 

  5. Wiginton JC (1980) A note on the comparison of logit and discriminant models of consumer credit behavior. J Financ Quant Anal 15:757–770

    Article  Google Scholar 

  6. Fisher RA (1936) The use of multiple measurements in taxonomic problems. Ann Eugen 7:179–188

    Article  Google Scholar 

  7. Glover F (1990) Improved linear programming models for discriminant analysis∗. Decis Sci 21:771–785

    Article  Google Scholar 

  8. Grablowsky BJ, Talley WK (1981) Probit and discriminant functions for classifying credit applicants. J Econ Bus 33:254–261

    Google Scholar 

  9. Henley W, Hand DJ (1996) A k-nearest-neighbour classifier for assessing consumer credit risk. The Statistician:77–95

  10. Van Gestel IT, Baesens B, Garcia IJ, Van Dijcke P (2003) A support vector machine approach to credit scoring. Bank en Financiewezen, pp 73–82

  11. Chen M-C, Huang S-H (2003) Credit scoring and rejected instances reassigning through evolutionary computation techniques. Expert Syst Appl 24:433–441

    Article  Google Scholar 

  12. Varetto F (1998) Genetic algorithms applications in the analysis of insolvency risk. J Bank Financ 22:1421–1439

    Article  Google Scholar 

  13. Malhotra R, Malhotra D (2003) Evaluating consumer loans using neural networks. Omega 31:83–96

    Article  Google Scholar 

  14. Smalz R, Conrad M (1994) Combining evolution with credit apportionment: a new learning algorithm for neural nets. Neural Netw 7:341–351

    Article  Google Scholar 

  15. Abdou H, Pointon J, El-Masry A (2008) Neural nets versus conventional techniques in credit scoring in Egyptian banking. Expert Syst Appl 35:1275–1292

    Article  Google Scholar 

  16. Angelini E, di Tollo G, Roli A (2008) A neural network approach for credit risk evaluation. Q Rev Econ Finance 48:733–755

    Article  Google Scholar 

  17. Khashman A (2010) Neural networks for credit risk evaluation: investigation of different neural models and learning schemes. Expert Syst Appl 37:6233–6239

    Article  Google Scholar 

  18. Lai KK, Yu L, Wang S, Zhou L (2006) Credit risk analysis using a reliability-based neural network ensemble model. In: Artificial neural networks–ICANN 2006. Springer, Berlin, pp 682–690

  19. Min JH, Lee Y-C (2008) A practical approach to credit scoring. Expert Syst Appl 35:1762–1770

    Article  Google Scholar 

  20. Huang G-B, Zhu Q-Y, Siew C-K (2006) Extreme learning machine: theory and applications. Neurocomputing 70:489–501

    Article  Google Scholar 

  21. Liu N, Wang H (2010) Ensemble based extreme learning machine. IEEE Signal Process Lett 17:754–757

    Article  Google Scholar 

  22. You Z-H, Lei Y-K, Zhu L, Xia J, Wang B (2013) Prediction of protein-protein interactions from amino acid sequences with ensemble extreme learning machines and principal component analysis. BMC Bioinf 14:S10

    Article  Google Scholar 

  23. Lian C, Zeng Z, Yao W, Tang H (2014) Ensemble of extreme learning machine for landslide displacement prediction based on time series analysis. Neural Comput & Applic 24:99–107

    Article  Google Scholar 

  24. Zhai J-H, Xu H-Y, Wang X-Z (2012) Dynamic ensemble extreme learning machine based on sample entropy. Soft Comput 16:1493–1502

    Article  Google Scholar 

  25. Tsoumakas G, Partalas I, Vlahavas I (2009) An ensemble pruning primer. In: Applications of supervised and unsupervised ensemble methods. Springer, Berlin, pp 1–13

  26. Partalas I, Tsoumakas G, Vlahavas I (2010) An ensemble uncertainty aware measure for directed hill climbing ensemble pruning. Mach Learn 81:257–282

    Article  MathSciNet  Google Scholar 

  27. Caruana R, Niculescu-Mizil A, Crew G, Ksikes A (2004) Ensemble selection from libraries of models. In: Proceedings of the 21st international conference on machine learning, p 18

  28. Partalas I, Tsoumakas G, Vlahavas I A study on greedy algorithms for ensemble pruning, Technical Report TR-LPIS-360-12, Department of Informatics, Aristotle University of Thessaloniki, Greece 2012

  29. Martinez-Muoz G, Hernández-Lobato D, Suárez A (2009) An analysis of ensemble pruning techniques based on ordered aggregation. IEEE Trans Pattern Anal Mach Intell 31:245–259

    Article  Google Scholar 

  30. Dai Q (2013) A novel ensemble pruning algorithm based on randomized greedy selective strategy and ballot. Neurocomputing 122:258–265

    Article  Google Scholar 

  31. Dai Q (2013) A competitive ensemble pruning approach based on cross-validation technique. Knowl-Based Syst 37:394–414

    Article  Google Scholar 

  32. Dai Q, Liu Z (2013) ModEnPBT: a modified backtracking ensemble pruning algorithm. Appl Soft Comput 13:4292–4302

    Article  Google Scholar 

  33. Liu Z, Dai Q, Liu N (2014) Ensemble selection by GRASP. Appl Intell 41:128–144

    Article  MathSciNet  Google Scholar 

  34. Banfield RE, Hall LO, Bowyer KW, Kegelmeyer WP (2005) Ensemble diversity measures and their application to thinning. Inf Fusion 6:49–62

    Article  Google Scholar 

  35. Margineantu DD, Dietterich TG (1997) Pruning adaptive boosting. In: ICML, pp 211–218

  36. Martýnez-Munoz G, Suárez A (2004) Aggregation ordering in bagging. In: Proceedings of the IASTED international conference on artificial intelligence and applications, pp 258–263

  37. Festa P, Pardalos PM, Pitsoulis LS, Resende MG (2007) GRASP with path relinking for the weighted MAXSAT problem. Journal of Experimental Algorithmics (JEA) 11:2.4

    MathSciNet  MATH  Google Scholar 

  38. Huang G-B (2003) Learning capability and storage capacity of two-hidden-layer feedforward networks. IEEE Trans Neural Netw 14:274–281

    Article  Google Scholar 

  39. Huang G-B, Babri HA (1998) Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions. IEEE Trans Neural Netw 9:224–229

    Article  Google Scholar 

  40. Rao CR, Mitra SK (1971) Generalized inverse of matrices and its applications, vol 7. Wiley, New York

    MATH  Google Scholar 

  41. Johnson CR (1990) Matrix theory and applications: american mathematical soc.

  42. Breiman L (1996) Bagging predictors. Mach Learn 24:123–140

    MathSciNet  MATH  Google Scholar 

  43. Feo TA, Resende MG (1995) Greedy randomized adaptive search procedures. J Glob Optim 6:109–133

    Article  MathSciNet  MATH  Google Scholar 

  44. Layeb A, Selmane M, Elhoucine MB (2013) A new greedy randomised adaptive search procedure for multiple sequence alignment. Int J Bioinforma Res Appl 9:323–335

    Article  Google Scholar 

  45. Glover F (1997) Tabu search and adaptive memory programming—advances, applications and challenges. In: Interfaces in computer science and operations research. Springer, Berlin, pp 1–75

  46. Gevezes T, Pitsoulis L (2013) A greedy randomized adaptive search procedure with path relinking for the shortest superstring problem. J Comb Optim:1–25

  47. Ribeiro CC, Resende MG (2012) Path-relinking intensification methods for stochastic local search algorithms. J Heuristics 18:193–214

    Article  Google Scholar 

  48. Martínez-Muñoz G, Suárez A (2006) Pruning in ordered bagging ensembles. In: Proceedings of the 23rd international conference on Machine learning, pp 609–616

  49. Asuncion A, Newman D UCI repository of machine learning databases. http://www.ics.uci.edu/~mlearn/MLRepository.html or ftp.ics.uci.edu:pub/machine-learning-databases

  50. Feo TA, Resende MG, Smith SH (1994) A greedy randomized adaptive search procedure for maximum independent set. Oper Res 42:860–878

    Article  MATH  Google Scholar 

  51. Lin CJ LIBSVM: a library for support vector machines. http://www.csie.ntu.edu.tw/~cjlin/

Download references

Acknowledgments

This work is supported by the National Natural Science Foundation of China under Grant no. 61473150.

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 210016, China

    Ting Zhang & Qun Dai

Authors
  1. Ting Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qun Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qun Dai.

Ethics declarations

Compliance with Ethical Standards

Compliance with Ethical Standards

  1. 1.

    Disclosure of potential conflict of interest

    Conflict of interest: The authors declare that they have no conflict of interest.

  2. 2.

    Research involving Human Participants and/or Animals

    All procedures performed in studies involving human participants were carried out in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

    For this type of study formal consent was not required.

    All applicable international, national, and/or institutional guidelines for the care and use of animals were followed.

  3. 3.

    Informed consent

Informed consent was obtained from all individual participants involved in the study.

Additional informed consent was obtained from all individual participants for whom identifying information is included in this article.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, T., Dai, Q. Hybrid ensemble selection algorithm incorporating GRASP with path relinking. Appl Intell 44, 704–724 (2016). https://doi.org/10.1007/s10489-015-0724-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-015-0724-4

Keywords

Search

Navigation