ALADIN: Algorithms for Learning and Architecture DetermINation

Abstract

There are many multi-layered neural networks which can be used in an application requiring a feed-forward neural network with a fixed number of input and output units. For a given training set, the generalization ability of a multi-layered neural network depends on the number of hidden layers as well as the number of hidden units per layer. There seems to be an agreement in the literature that a neural network with at most two layers of hidden units can approximate any given set of functions, provided that there are enough units per hidden layer (Lapedes and Farber, 1988; Hertz et al., 1991). In addition, it has been shown that a neural network with one layer of enough hidden units can approximate any set of continuous functions (Cybenko, 1989). However, the number of units per hidden layer which guarantees satisfactory performance of the network after training remains an open problem. Some theoretical lower and upper bounds on the sample size versus the size of a network with one layer of hidden units needed for valid generalization have been reported in (Baum and Haussier, 1989). However, there are no reliable criteria for the selection of the number of hidden units per layer required by neural networks with multiple hidden layers. At this stage of research, the selection of the number of hidden layers and the number of hidden units per layer is more an art than a science. In practice, the selection of multi-layered neural networks is usually performed by experimenting with networks of various sizes until the desired network performance is reached. The lack of a procedure for determining the multi-layered network which achieves the best performance in a given application is considered to be one of the major limitations of feed-forward neural networks. This criticism of feed-forward neural networks is fair, since selection of an inadequate network can considerably degrade the performance of the resulting system.