Multi-layer self-organizing polynomial neural networks and their development with the use of genetic algorithms

https://doi.org/10.1016/j.jfranklin.2005.09.005Get rights and content

Abstract

In this paper, we introduce a new architecture of genetic algorithms (GA)-based self-organizing polynomial neural networks (SOPNN) and discuss a comprehensive design methodology. Let us recall that the design of the “conventional” PNNs uses an extended group method of data handling (GMDH) and exploits polynomials (such as linear, quadratic, and modified quadratic functions) as well as considers a fixed number of input nodes (as being selected in advance by a network designer) at polynomial neurons (or nodes) located in each layer. The proposed GA-based SOPNN gives rise to a structurally optimized structure and comes with a substantial level of flexibility in comparison to the one we encounter in conventional PNNs. The design procedure applied in the construction of each layer of a PNN deals with its structural optimization involving the selection of preferred nodes (or PNs) with specific local characteristics (such as the number of input variables, the order of the polynomial, and a collection of the specific subset of input variables) and addresses specific aspects of parametric optimization. An aggregate performance index with a weighting factor is proposed in order to achieve a sound balance between approximation and generalization (predictive) abilities of the network. To evaluate the performance of the GA-based SOPNN, the model is experimented with using chaotic time series data. A comparative analysis reveals that the proposed GA-based SOPNN exhibits higher accuracy and superb predictive capability in comparison to some previous models available in the literature.

Introduction

While neural networks, fuzzy sets and evolutionary computing have expanded and enriched a field of modeling quite immensely, they have also gave rise to a number of new methodological issues and increased our awareness about tradeoffs one has to make in system modeling [1]. When the dimensionality of the model goes up (say, the number of variables increases), so do the difficulties. In particular, when dealing with high-order nonlinear and multivariable equations of the model, we require a vast amount of data for estimating all its parameters [1].

To help alleviate the problems, one of the first approaches along the line of a systematic design of nonlinear relationships between system's inputs and outputs comes under the name of a group method of data handling (GMDH) [2]. The GMDH algorithm generates an optimal structure of the model through successive generations of partial descriptions of data (PDs) being regarded as quadratic regression polynomials of two input variables. While providing with a systematic design procedure, GMDH comes with some drawbacks. First, it tends to generate quite complex polynomials even for relatively simple systems (experimental data). Second, owing to its limited generic structure (that is quadratic two-variable polynomials), GMDH also tends to produce an overly complex network (model) when it comes to highly nonlinear systems. Third, if there are less than three input variables, GMDH algorithm does not generate a highly versatile structure.

To alleviate the problems associated with the GMDH, polynomial neural networks (PNN) were introduced by Oh et al. [3], [4], [14]. They can be viewed as a new category of neural networks. In a nutshell, these networks come with a high level of flexibility associated with each node (processing element forming a partial description (PD) (or polynomial neurons (PN))), can have a different number of input variables as well as exploit different orders of the polynomials used in the nodes (say, linear, quadratic, modified quadratic, etc.). Although the PNN has a flexible architecture whose potential can be fully utilized through a systematic design, it is difficult to obtain the structurally and parametrically optimized network because of the limited design of the PNs located in each layer of the PNN. In other words, when we construct PNs of each layer in the conventional PNN, such parameters as the number of input variables (nodes), the order of the polynomial, and the input variables available within a PN are fixed (selected) in advance by the designer. Accordingly, the PNN algorithm exhibits some tendency to produce overly complex networks as well as a repetitive computation load by the trial and error method and/or a repetitive parameter adjustment by designer like in case of the original GMDH algorithm. In order to generate a structurally and parametrically optimized network, such parameters need to be optimal.

In this study, in addressing the above problems with the conventional PNN as well as the GMDH algorithm, we introduce a new genetic design approach; as a consequence we will be referring to these networks as GA-based self-organizing polynomial neural networks (SOPNN). The determination of the optimal values of the parameters available within an individual PN (viz. the number of input variables, the order of the polynomial, and input variables) leads to a structurally and parametrically optimized network. As a result, this network is more flexible as well as exhibits simpler topology in comparison to the conventional PNNs discussed in the previous research. In the development of the network, we introduce an aggregate objective function (performance index) that deals with training data and testing data, and elaborate on its optimization to produce a meaningful balance between approximation and generalization abilities of the network [5].

To evaluate the performance of the proposed model, we exploit well-known time series data [6], [7], [8], [9], [10], [11].

Section snippets

Polynomial neural networks

In this section, we elaborate on the architecture and a development process of the conventional PNN. First, we briefly discuss polynomial neurons forming a basic computing node of the network. Next, the structural and parametric optimization of PNNs is described with a special emphasis placed on the growth process of the network.

The algorithms and design procedure of GA-based SOPNN

Here we describe in detail the design method of the GA-based SOPNN. It entails a series of steps:

Step 1: Determine system's input variables.

Define system's input variables xi (i=1, 2,…, n) related to the output variable y. If required, the normalization of input data is carried out as well.

Step 2: Form training and testing data.

The input–output data set (xi, yi)=(x1i, x2i,…, xni, yi), i=1, 2,…, N (with N being the total number of data points) is divided into two parts, that is, a training and

Simulation studies

In this section, we demonstrate how the GA-based SOPNN can be utilized to predict future values of a chaotic Mackey–Glass time series. The performance of the network is also contrasted with some other models existing in the literature [6], [7], [8], [9], [10], [11]. The prediction of future values of this series arises as a benchmark problem that has been used and reported by a number of researchers. From the Mackey–Glass time series x(t), we extracted 1000 input–output data pairs in the

Conclusions

In this study, we investigated a class of GA-based self-organizing polynomial neural networks (SOPNN). The GA-based design procedure applied at each stage (layer) of the SOPNN leads to the selection of some preferred nodes (or PNs) with local characteristics (such as the number of input variables, the order of the polynomial, and input variables) available within the PNN. These design alternatives contribute to the flexibility of the resulting architecture of the network. The underlying

Acknowledgements

This work has been supported by KESRI (R-2004-B-274) funded by MOCIE (Ministry of Commerce, Industry and Energy).

References (14)

There are more references available in the full text version of this article.

Cited by (0)

View full text