Learning in neural networks by using tangent planes to constraint surfaces

doi:10.1016/0893-6080(93)90005-H

Neural Networks

Volume 6, Issue 3, 1993, Pages 385-392

https://doi.org/10.1016/0893-6080(93)90005-H Get rights and content

Abstract

The principal disadvantage of the back propagation gradient descent learning algorithm for multilayer feedforward neural networks is its relatively slow rate of convergence. An alternative method, adjusting weights by moving to the tangent planes to constraint surfaces, is shown to give significantly faster convergence whilst preserving the system of back propagating errors through the network.

References (7)

N. Baba
A new approach for finding the global minimum of error function of neural networks
Neural Networks
(1989)
R.A. Jacobs
Increased rates of convergence through learning rate adaptation
Neural Networks
(1988)
R. Battiti
Accelerated backpropagation learning: Two optimisation methods
Complex Systems
(1989)

There are more references available in the full text version of this article.

Cited by (4)

Learning in fully recurrent neural networks by approaching tangent planes to constraint surfaces
2012, Neural Networks
Citation Excerpt :
Finally, the experimental results are given in Section 4. Lee (1993, 1997) proposed an algorithm for supervised learning in multilayered feedforward neural networks which gives significantly faster convergence than the gradient descent back-propagation algorithm. This tangent plane algorithm treats each teaching value as a constraint which defines a surface in the weight space of the network.
In this paper we present a new variant of the online real time recurrent learning algorithm proposed by Williams and Zipser (1989). Whilst the original algorithm utilises gradient information to guide the search towards the minimum training error, it is very slow in most applications and often gets stuck in local minima of the search space. It is also sensitive to the choice of learning rate and requires careful tuning. The new variant adjusts weights by moving to the tangent planes to constraint surfaces. It is simple to implement and requires no parameters to be set manually. Experimental results show that this new algorithm gives significantly faster convergence whilst avoiding problems like local minima.
A transformation strategy for implementing distributed, multilayer feedforward neural networks: Backpropagation transformation
1997, Future Generation Computer Systems
Most artificial neural networks (ANNs) have a fixed topology during learning, and often suffer from a number of shortcomings as a result. Variations of ANNs that use dynamic topologies have shown ability to overcome many of these problems. This paper introduces location-independent transformations (LITs) as a general strategy for implementing distributed feed forward networks that use dynamic topologies (dynamic ANNs) efficiently in parallel hardware. A LIT creates a set of location-independent nodes, where each node computes its part of the network output independent of other nodes, using local information. This type of transformation allows efficient support for adding and deleting nodes dynamically during learning. In particular, this paper presents a LIT that supports both the standard (static) multilayer backpropagation network, and backpropagation with dynamic extensions. The complexity of both learning and execution algorithms is O(q(Nlog M)) for a single pattern, where q is the number of weight layers in the original network, N the number of nodes in the widest node layer in the original network, and M is the number of nodes in the transformed network (which is linear in the number hidden nodes in the original network). This paper extends previous work with 2-weight-layer backpropagation networks.
Training feedforward neural networks: An algorithm giving improved generalization
1997, Neural Networks
Deterministic neural classification
2008, Neural Computation

View full text

Neural Networks

Original contributionLearning in neural networks by using tangent planes to constraint surfaces

Abstract

A new approach for finding the global minimum of error function of neural networks

Neural Networks

Increased rates of convergence through learning rate adaptation

Neural Networks

Accelerated backpropagation learning: Two optimisation methods

Complex Systems

Learning in fully recurrent neural networks by approaching tangent planes to constraint surfaces

A transformation strategy for implementing distributed, multilayer feedforward neural networks: Backpropagation transformation

Training feedforward neural networks: An algorithm giving improved generalization

Deterministic neural classification

Original contribution
Learning in neural networks by using tangent planes to constraint surfaces