Abstract
To solve complex problems such as multi-input function approximation by using neural networks and to overcome the inherent defects of traditional back-propagation neural networks, a single hidden-layer feed-forward sine-activated neural network, sine neural network (SNN), is proposed and investigated in this paper. Then, a double-stage weights and structure determination (DS-WASD) method, which is based on the weights direct determination method and the approximation theory of using linearly independent functions, is developed to train the proposed SNN. Such a DS-WASD method can efficiently and automatically obtain the relatively optimal SNN structure. Numerical results illustrate the validity and efficacy of the SNN model and the DS-WASD method. That is, the proposed SNN model equipped with the DS-WASD method has great performance of approximation on multi-input function data.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bartelett PL (1998) The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network. IEEE Trans Inf Theory 44(2):525–536
Cheney W, Lignt W (2000) A course in approximation theory. Amer Math Society, Washington, DC
Cybenko G (1989) Approximation by superpositions of a sigmoidal function. Math Control Signals Syst 2(4):303–314
Deimling K (1985) Nonlinear functional analysis. Springer, Berlin
Friedman JH (2001) Greedy function approximation: a gradient boosting machine. Ann Stat 29(5):1189–1232
Funahashi K (1989) On the approximate realization of continuous mappings by neural networks. Neural Netw 2(3):183–192
Halawa K (2011) A method to improve the performance of multilayer perception by utilizing various activation functions in the last hidden layer and the least squares method. Neural Process Lett 34(2):293–303
Ham FM, Kostanic I (2001) Principles of neurocomputing for science and engineering. McGraw-Hill Companies, New York
Hornik H (1991) Approximation capabilities of multilayer feedforward networks. Neural Netw 4(2):251–257
Huang GB, Chen L (2006) Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans Neural Netw 17(4):879–892
John M, Kurtis F (2004) Numerical methods using MATLAB, 4th edn. Prentice Hall, London
Jones LK (1990) Constructive approximations for neural networks by sigmoidal functions. IEEE Proc 78(10):1586–1589
Kadirkamanathan V, Niranjan M (1993) A function estimation approach to sequential learning with neural networks. Neural Comput 5(6):954–975
Li YW, Sundararajan N, Saratchandran P (1997) A sequential learning scheme for function approximation using minimal radial basis function neural networks. Neural Comput 9(2):461–478
Llanas B, Sainz FJ (2006) Constructive approximate interpolation by neural networks. J Comput Appl Math 188(2):283–308
Mahil J, Raja TSR (2014) An intelligent biological inspired evolutionary algorithm for the suppression of incubator interference in premature infants ECG. Soft Comput 18(3):571–578
Moré JJ (1978) The Levenberg-Marquardt algorithm: implementation and theory. Lect Notes Math 630:105–116
Pérez-Cruz F, Camps-Valls G, Soria-Olivas E, Pérez-Ruixo JJ, Figueiras-Vidal AR, Artés-Rodríguez A (2002) Multi-dimensional function approximation and regression estimation. Lect Notes Comput Sci 2415:796–796
Romero E, Alquézar R (2002) A new incremental method for function approximation using feed-forward neural networks. In: Proceedings of the international joint conference on neural networks. Honolulu, American, pp 1968–1973
Sheela KG, Deepa SN (2014) Performance analysis of modeling framework for prediction in wind farms employing artificial neural networks. Soft Comput 18(3):607–615
Steven JL (1999) Linear algebra with applications, 5th edn. Prentice Hall/Pearson, New Jersey
Taylor JG (1993) Mathematical approaches to neural networks. Elsevier Science Publishers, The Netherlands
Wang GT, Li P, Cao JT (2012) Variable activation function extreme learning machine based on residual prediction compensation. Soft Comput 16(9):1477–1484
Wang JJ, Xu ZB (2009) Approximation method of multivariate polynomials by feedforward neural networks. Chin J Comput 32(12):2482–2488
Zhang Y, Tan N (2010) Weights direct determination of feedforward neural networks without iteration BP-training. In: Wang LSL, Hong TP (eds) Intelligent soft computation and evolving data mining: integrating advanced technologies. IGI Global, USA
Acknowledgments
This work is supported by the 2012 Scholarship Award for Excellent Doctoral Student Granted by Ministry of Education of China (with number 3191004), by the Guangdong Provincial Innovation Training Program for University Students (with number 1055813063), and by the Foundation of Key Laboratory of Autonomous Systems and Networked Control of Ministry of Education of China (with number 2013A07). Besides, the authors would like to thank the editor and reviewers sincerely for their time and effort spent in handling the paper, as well as the detailed and constructive comments provided for improving further the presentation and quality of the paper.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by V. Loia.
Appendix: comparison with sigmoid-activated neural network based on LM algorithm
Appendix: comparison with sigmoid-activated neural network based on LM algorithm
For comparison and for further illustration of the efficacy of the SNN equipped with the DS-WASD method, in this appendix, the 16 objective functions are tested by using the traditional BP iteration method; specifically, using the sigmoid-activated neural network (or say, sigmoid neural network) based on LM (Levenberg–Marquardt) algorithm. Note that, for fair comparison with existing BP iteration algorithms, we choose a relatively good (or say, iteratively best) one, i.e., the LM algorithm which is of second order (Bartelett 1998; Ham and Kostanic 2001; Mahil and Raja 2014; Moré 1978; Taylor 1993).
In the numerical studies, all the training and testing samples are chosen to be the same as those used in Sects. 3 and 4. In addition, the desired training MSE (as a criterion) for each objective function is set to be the training MSE shown in Table 3 (which is obtained by using the SNN equipped with the DS-WASD method). As aforementioned, the activation function of each hidden-layer neuron in the sigmoid neural network is the widely-used sigmoid function. The corresponding numerical results are thus shown in Table 5 and discussed as follows.
First, with the runtime limitation of 600 seconds (s), only objective functions 5 through 8 (completely) and 9 (partially) reach the corresponding accuracies of the desired training MSE. Second, most testing MSEs are worse than those in Table 3, which shows that the generalization ability of the sigmoid neural network based on the LM algorithm is relatively weaker than that of the proposed SNN equipped with the DS-WASD method. Third, for different objective functions, the relatively optimal numbers of hidden-layer neurons are different, which indicates that we have to try a lot to determine the final/optimal neural-network structure by using traditional BP iteration method. In summary, it is evident that the performance of the SNN equipped with the DS-WASD method is better than that of the original sigmoid-activated neural network.
Rights and permissions
About this article
Cite this article
Zhang, Y., Qu, L., Liu, J. et al. Sine neural network (SNN) with double-stage weights and structure determination (DS-WASD). Soft Comput 20, 211–221 (2016). https://doi.org/10.1007/s00500-014-1491-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-014-1491-6