Gradient descent learning for quaternionic Hopfield neural networks

Abstract

A Hopfield neural network (HNN) is a neural network model with mutual connections. A quaternionic HNN (QHNN) is an extension of HNN. Several QHNN models have been proposed. The hybrid QHNN utilizes the non-commutativity of quaternions. It has been shown that hybrid QHNNs with the Hebbian learning rule outperformed QHNNs in noise tolerance. The Hebbian learning rule, however, is a primitive learning algorithm, and it is necessary that we study advanced learning algorithms. Although the projection rule is one of a few promising learning algorithms, it is restricted by network topologies and cannot be applied to hybrid QHNNs. In the present work, we propose gradient descent learning, which can be applied to hybrid QHNNs. We compare the performance of gradient descent learning with that of projection rule. Results showed that the gradient descent learning outperformed projection rule in noise tolerance in computer simulations. For small training-pattern sets, hybrid QHNNs with gradient descent learning produced the best performances. QHNNs did so for large training-pattern set. In future, gradient descent learning will be extended to QHNNs with different network topology and activation function.

Introduction

Models of neural networks have been extended and applied in a variety of areas. Complex-valued neural networks have been one of the most successful extensions. Neural networks have currently been extended to quaternionic versions, such as multi-layered perceptrons and Hopfield neural networks (HNNs) [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12]. Moreover, quaternions have often been used in signal processing [13], [14].

An HNN is a recurrent neural network with mutual connections, and is one of the most successful neural network models. Complex-valued HNNs can represent phasor information, and have been applied to the storage of gray-scale images [15], [16]. To improve storage capacity and noise tolerance, several learning algorithms have been studied [17], [18], [19]. The projection rule is a fast learning algorithm, but is restricted by network topologies [20], [21], [22]. The gradient descent learning algorithm is iterative and flexible in network topologies, although its learning speed is slower [23], [24], [25].

Complex-valued HNNs have been extended to quaternionic HNNs (QHNNs). Several types of activation functions have been proposed for QHNNs. The split activation function is the simplest [26]. The multistate activation function has been applied to the storage of color images [27]. Attractors of QHNNs have also been studied, and a variety of network topologies have been proposed [28]. Bidirectional models with multistate activation function were found to improve noise tolerance [29]. A hybrid QHNN is a model with the split activation function. It improves noise tolerance by utilizing the non-commutativity of quaternions [30]. Thus far, the Hebbian learning rule and the projection rule have been studied. The Hebbian learning rule, however, is primitive, with very little storage capacity [30]. The projection rule, meanwhile, cannot be applied to hybrid QHNNs, because it requires fully left- or right-connections in QHNNs. In this study, we propose the gradient descent learning. Gradient descent learning is flexible in network topologies, and can be applied to hybrid QHNNs. Signal processing provides a novel technique, the GHR calculus [31], [32], [33]. Using the GHR calculus, gradient descent learning can be represented in an elegant form. We performed computer simulations to analyze storage capacity and noise tolerance. We show that either the QHNNs or hybrid QHNNs with gradient descent learning outperformed the QHNNs with projection rule.

This paper is organized as follows. Section 3 describes quaternions and QHNNs. In Section 3, gradient descent learning is provided. Section 5 describes the computer simulation and Section 5 discusses the results. Finally, Section 6 presents the conclusion.