An Analysis of the Delta Rule

Abstract

The delta rule and its generalisations is one of the most practically useful learning algorithms for connectionist systems. (See [5] and, e.g. [1]). It is often stated that the method is gradient descent for the least squares error. However, neither part of this statement is true for finitely large learning rates. The purpose of this paper is to employ certain mathematical tools, largely borrowed from numerical analysis, to provide a rigourous framework for discussing the behaviour of learning algorithms. To this end, a more or less complete analysis of the original linear version of the delta rule is presented. It is shown that when the rule is applied by repetitively cycling through a fixed epoch of patterns, updating the weights after each pattern, the algorithm generates a limit cycle. The least squares error of this limit cycle appoaches that of the true minimum quadratically as the learning rate tends to zero. The algorithm is convergent and numerically stable subject only to a simple normalisation condition. By contrast, if the weights are updated after the complete epoch of patterns has been presented, the iteration has a fixed point which is the true least squares minimum. However the algorithm may have very bad numerical stability and convergence properties, even for problems which are “good” from the point of view of learning. This simple linear case is of limited practical use, but heuristic and numerical evidence suggests that the analysis does give insight into the behaviour of the method for more useful cases such as back propagation networks. Current work is directed to a rigorous justification of this. It is the author’s belief that the methods can be extended to other learning paradigms.