Abstract
The performance of the support vector regression (SVR) model is sensitive to the kernel type and its parameters. The determination of an appropriate kernel type and the associated parameters for SVR is a challenging research topic in the field of support vector learning. In this study, we present a novel method for simultaneous optimization of the SVR kernel function and its parameters, formulated as a mixed integer optimization problem and solved using the recently proposed heuristic ‘extremal optimization (EO)’. We present the problem formulation for the optimization of the SVR kernel and parameters, the EO-SVR algorithm, and experimental tests with five benchmark regression problems. The results of comparison with other traditional approaches show that the proposed EO-SVR method provides better generalization performance by successfully identifying the optimal SVR kernel function and its parameters.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ali, S., Smith, K.A., 2006. A meta-learning approach to automatic kernel selection for support vector machines. Neurocomputing, 70(1–3):173–186. [doi:10.1016/j.neucom.2006.03.004]
Avci, E., 2009. Selecting of the optimal feature subset and kernel parameters in digital modulation classification by using hybrid genetic algorithm—support vector machines: HGASVM. Expert Syst. Appl., 36(2):1391–1402. [doi:10.1016/j.eswa.2007.11.014]
Bak, P., Sneppen, K., 1993. Punctuated equilibrium and criticality in a simple model of evolution. Phys. Rev. Lett., 71(24):4083–4086. [doi:10.1103/PhysRevLett.71.4083]
Bak, P., Tang, C., Wiesenfeld, K., 1987. Self-organized criticality: an explanation of the 1/f noise. Phys. Rev. Lett., 59(4):381–384. [doi:10.1103/PhysRevLett.59.381]
Boettcher, S., Percus, A.G., 1999. Extremal Optimization: Methods Derived from Co-evolution. Proc. Genetic and Evolutionary Computation Conf., p.825–832.
Boettcher, S., Percus, A., 2000. Nature’s way of optimizing. Artif. Intell., 119(1–2):275–286. [doi:10.1016/S0004-3702 (00)00007-2]
Burges, C.J.C., 1998. A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Discov., 2(2):121–167. [doi:10.1023/A:1009715923555]
Chen, M.R., Lu, Y.Z., 2008. A novel elitist multiobjective optimization algorithm: multiobjective extremal optimization. Eur. J. Oper. Res., 188(3):637–651. [doi:10.1016/j.ejor.2007.05.008]
Chen, M.R., Lu, Y.Z., Yang, G.K., 2007. Multiobjective extremal optimization with applications to engineering design. J. Zhejiang Univ.-Sci. A, 8(12):1905–1911. [doi:10.1631/jzus.2007.A1905]
Chen, Y.W., Lu, Y.Z., Chen, P., 2007. Optimization with extremal dynamics for the traveling salesman problem. Phys. A, 385(1):115–123. [doi:10.1016/j.physa.2007.06.014]
Chuang, C.C., Lee, Z.J., 2011. Hybrid robust support vector machines for regression with outliers. Appl. Soft. Comput., 11(1):64–72. [doi:10.1016/j.asoc.2009.10.017]
Engelbrecht, A.P., 2007. Computational Intelligence: an Introduction (2nd Ed.). John Wiley & Sons, New York.
Friedrichs, F., Igel, C., 2005. Evolutionary tuning of multiple SVM parameters. Neurocomputing, 64(1–4):107–117. [doi:10.1016/j.neucom.2004.11.022]
Hou, S.M., Li, Y.R., 2009. Short-term fault prediction based on support vector machines with parameter optimization by evolution strategy. Expert Syst. Appl., 36(10):12383–12391. [doi:10.1016/j.eswa.2009.04.047]
Howley, T., Madden, M.G., 2005. The genetic kernel support vector machine: description and evaluation. Artif. Intell. Rev., 24(3–4):379–395. [doi:10.1007/s10462-005-9009-3]
Hsu, C.W., Chang, C.C., Lin, C.J., 2004. A Practical Guide to Support Vector Classification. Technical Report, Department of Computer Science and Information Engineering, National Taiwan University.
Jeng, J.T., 2006. Hybrid approach of selecting hyperparameters of support vector machine for regression. IEEE Trans. Syst. Man Cybern. B, 36(3):699–709. [doi:10.1109/TSMCB.2005.861067]
Lorena, A.C., de Carvalho, A., 2008. Evolutionary tuning of SVM parameter values in multiclass problems. Neurocomputing, 71(16–18):3326–3334. [doi:10.1016/j.neucom.2008.01.031]
Lu, Y.Z., Chen, M.R., Chen, Y.W., 2007. Studies on Extremal Optimization and Its Applications in Solving Real World Optimization Problems. IEEE Symp. on Foundations of Computational Intelligence, p.162–168. [doi:10.1109/FOCI.2007.372163]
Mao, Y., Zhou, X., Pi, D., Sun, Y., Wong, S.T.C., 2005. Parameters selection in gene selection using Gaussian kernel support vector machines by genetic algorithm. J. Zhejiang Univ.-Sci., 6B(10):961–973. [doi:10.1631/jzus.2005.B0961]
Min, J.H., Lee, Y.C., 2005. Bankruptcy prediction using support vector machine with optimal choice of kernel function parameters. Expert Syst. Appl., 28(4):603–614. [doi:10.1016/j.eswa.2004.12.008]
Pai, P.F., Hong, W.C., 2005. Support vector machines with simulated annealing algorithms in electricity load forecasting. Energy Conv. Manag., 46(17):2669–2688. [doi:10.1016/j.enconman.2005.02.004]
Qiao, J.F., Wang, H.D., 2008. A self-organizing fuzzy neural network and its applications to function approximation and forecast modeling. Neurocomputing, 71(4–6):564–569. [doi:10.1016/j.neucom.2007.07.026]
Saini, L.M., Aggarwal, S.K., Kumar, A., 2010. Parameter optimisation using genetic algorithm for support vector machine-based price-forecasting model in National electricity market. IET Gener. Transm. Distr., 4(1):36–49. [doi:10.1049/iet-gtd.2008.0584]
Steve, G., 1998. Support Vector Machines Classification and Regression. ISIS Technical Report, Image, Speech, & Intelligent Systems Group, University of Southampton, UK.
Tang, X., Zhuang, L., Jiang, C., 2009. Prediction of silicon content in hot metal using support vector regression based on chaos particle swarm optimization. Expert Syst. Appl., 36(9):11853–11857. [doi:10.1016/j.eswa.2009.04.015]
Thadani, K., Ashutosh, Jayaraman, V.K., Sundararajan, V., 2006. Evolutionary Selection of Kernels in Support Vector Machines. Int. Conf. on Advanced Computing and Communications, p.19–24. [doi:10.1109/ADCOM.2006.4289849]
Vapnik, V., 1995. The Nature of Statistical Learning Theory. Springer-Verlag, New York.
Wu, C.H., Tzeng, G.H., Goo, Y.J., Fang, W.C., 2007. A real-valued genetic algorithm to optimize the parameters of support vector machine for predicting bankruptcy. Expert Syst. Appl., 32(2):397–408. [doi:10.1016/j.eswa.2005.12.008]
Wu, C.H., Tzeng, G.H., Lin, R.H., 2009. A novel hybrid genetic algorithm for kernel function and parameter optimization in support vector regression. Expert Syst. Appl., 36(3):4725–4735. [doi:10.1016/j.eswa.2008.06.046]
Wu, Q., 2010. A hybrid-forecasting model based on Gaussian support vector machine and chaotic particle swarm optimization. Expert Syst. Appl., 37(3):2388–2394. [doi:10.1016/j.eswa.2009.07.057]
Zhang, L., Zhou, W., Jiao, L., 2004. Wavelet support vector machine. IEEE Trans. Syst. Man Cybern. B, 34(1):34–39. [doi:10.1109/TSMCB.2003.811113]
Zhang, X.L., Chen, X.F., He, Z.J., 2010. An ACO-based algorithm for parameter optimization of support vector machines. Expert Syst. Appl., 37(9):6618–6628. [doi:10.1016/j.eswa.2010.03.067]
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Chen, P., Lu, Yz. Extremal optimization for optimizing kernel function and its parameters in support vector regression. J. Zhejiang Univ. - Sci. C 12, 297–306 (2011). https://doi.org/10.1631/jzus.C1000110
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/jzus.C1000110
Key words
- Support vector regression (SVR)
- Extremal optimization (EO)
- Parameter optimization
- Kernel function optimization