Broad echo state network for multivariate time series prediction

doi:10.1016/j.jfranklin.2019.01.027

Journal of the Franklin Institute

Volume 356, Issue 9, June 2019, Pages 4888-4906

https://doi.org/10.1016/j.jfranklin.2019.01.027 Get rights and content

Abstract

In this paper, a broad echo state network with multiple reservoirs in parallel configuration (Broad-ESN) is proposed for a class of multivariate time series prediction. Firstly, through the unsupervised learning algorithm of restricted Boltzmann machine (RBM), the number of reservoirs of Broad-ESN can be determined, such that the dynamic characteristics of a class of multivariate time series can be fully reflected. Secondly, a parameter optimization method based on Davidon–Fletcher–Powell (DFP) quasi-Newton algorithm is proposed to optimize the reservoir parameters of Broad-ESN. Meanwhile, an output weights learning method based on output error is given to train the output weights of Broad-ESN. Thirdly, a sufficient condition for the echo state property of Broad-ESN is given. Finally, four examples are given to verify the effectiveness of Broad-ESN.

Introduction

Echo state network (ESN) [1], [2] is a class of recurrent neural network (RNN) [3], [4], whose hidden layer is replaced by a dynamical reservoir. Compared with RNN, the advantage of ESN is that only the output weights need to be trained, while the reservoir internal weights and input weights are given randomly. Nowadays, ESN has been paid much attention in many input-driven applications, for example, time-series prediction and classification [5], [6], [7], [8], [9], [10], [11], [12], dynamic pattern recognition [13], [14], system modeling or identification [15], [16], [17], [18], filtering or control [19], [20], [21], big data applications [22], [23], etc.

For the multivariate time series [24], [25], [26], due to the increasing of feature information, the traditional ESN can not meet the requirements of prediction performance. Therefore, many improved ESNs have been presented [5], [6], [12], [16], [17], [27], for example, in [12], a new model called adaptive elastic ESN (AEESN) is proposed to overcome the collinearity problem and a sparse solution for multivariate time series is obtained. In [6], a fast subspace decomposition echo state network (FSDESN) is given to solve the ill-posed problem of multivariate time series prediction. In [27], a multi-reservoir Echo State Network based on Sparse Bayesian method (MrBESN) is proposed to solve the problem of selecting reservoir weights. In order to obtain the higher prediction accuracy of multivariate time series, the reservoir sizes of these improved ESNs are usually larger. However, how to use a smaller size reservoir to achieve the higher prediction accuracy is an interesting work.

In addition, for the reservoir parameter optimization problem, the batch gradient descent (BGD) algorithm [5] is usually used to optimize the reservoir parameters of the existing ESNs. When optimization problem is a multi-peak optimization problem, the BGD algorithm may be trapped into the local minimum. For the output weights training problem, the matrix pseudo-inverse method [5], [6], [7] and gradient-based learning method [28] are usually used to train the output weights of the existing ESNs. However, the matrix pseudo-inverse method could cause a significant delay, such that a larger calculation error of computing prediction accuracy could be introduced. In [29], the authors claimed that the trained output weights by the gradient-based learning method could suffer from the danger of trapping into the local minima.

In order to overcome the aforementioned problems, considering the decomposition mechanism of the traditional reservoir, a novel echo state network with multiple reservoirs in parallel configuration, called broad echo state network (Broad-ESN), is proposed for a class of multivariate time series prediction. Compared with the existing ESNs, the dynamic feature of multivariate time series can be adequately reflected by virtue of the multi-reservoir of Broad-ESN. However, for the Broad-ESN, how to determine the number of reservoirs is a difficult problem. The existing literature does not give the corresponding selection rule. In order to determine the number of reservoirs, a restricted Boltzmann machine (RBM) [30], [31] is introduced into the Broad-ESN. Through the unsupervised learning algorithm of RBM, the feature of input information can be extracted, and thus the number of reservoirs will be determined. Because the reservoir parameters of Broad-ESN are significantly increased, the reservoir optimization problem will be transformed into multi-peak optimization problem. Thus, a new optimization method based on the Davidon–Fletcher–Powell (DFP) quasi-Newton algorithm [32], [33] is given to optimize the reservoir parameters of Broad-ESN.

The main contributions of this paper are as follows:

1.
A Broad-ESN is proposed for a class of multivariate time series prediction. Through the unsupervised learning algorithm of RBM, the number of reservoirs of Broad-ESN can be determined, such that the dynamic characteristics of a class of multivariate time series can be fully reflected.
2.
A parameter optimization method based on DFP quasi-Newton algorithm is proposed to optimize the reservoir parameters of Broad-ESN.
3.
An output weights learning method based on output error is given to train the output weights of Broad-ESN. In addition, a sufficient condition for the echo state property of Broad-ESN is given.

The remaining part of this paper is organized as follows. In Section 2, a Broad-ESN with multi-reservoir in parallel configuration is introduced. In Section 3, a parameter optimization method based on DFP quasi-Newton algorithm is given, and an output weights training method based on output error is given. Simulation examples are performed in Section 4. Finally, the conclusion and future work are given in Section 5.

Section snippets

Broad echo state network

In this section, we give an introduction to the multivariate time series prediction method based on Broad-ESN. In Section 2.1, the basic description of ESN is introduced. In Section 2.2, the basic concept of RBM is introduced. In Section 2.3, the proposed prediction model based on Broad-ESN is introduced. The echo state property of Broad-ESN is proved in Section 2.4.

Learning algorithm of reservoir parameters and output weights

In this section, some learning algorithms are given to train the Broad-ESN. In Section 3.1, a parameter optimization method based on DFP quasi-Newton algorithm is given. An output weights learning method based on output error is given in Section 3.2.

Simulation examples

In this section, the Broad-ESN with DFP quasi-Newton algorithm (27)–(32) and output weight learning method (33) is used for multivariate time series prediction. Two benchmark datasets and two real-world datasets are selected to verify the prediction performance of the Broad-ESN. The performance metric is the root mean squared errors (RMSE) of one-step ahead prediction. The definition of RMSE is given as follows: $\begin{matrix} R M S E = \sqrt{\sum_{n = 1}^{T} {(y (n) - d (n))}^{2} / (T - 1)} \end{matrix}$ where T is the number of data points in the sample

Conclusion

In this paper, a Broad-ESN is proposed for a class of multivariate time series prediction. According to the unsupervised learning algorithm of RBM, the number of reservoirs of Broad-ESN can be determined. A parameter optimization method based on DFP quasi-Newton algorithm is given to optimize the reservoir parameters of Broad-ESN. Meanwhile, an output weights learning method is given to train the output weights of Broad-ESN. Compared with ESN, Leaky-ESN, AEESN, RVESN and MrBESN, the simulation

Acknowledgment

This work was supported in part by the National Natural Science Foundation of China under Grants 61473070, 61433004, 61627809, and 61773074, and in part by the Fundamental Research Funds for the State Key Laboratory of Synthetical Automation for Process Industries (SAPI) under Grant 2018ZCX22, and in part by the Fundamental Research Funds for the Central Universities under Grant N160406002.

References (37)

H. Jaeger et al.
Optimization and applications of echo state networks with leaky-integrator neurons
Neural Netw.
(2007)
Q.L. Ma et al.
Functional echo state network for time series classification
Inf. Sci.
(2016)
G. Shi et al.
Energy consumption prediction of office buildings based on echo state networks
Neurocomputing
(2016)
L.H. Shen et al.
A novel echo state network for multivariate and nonlinear time series prediction
Appl. Soft Comput.
(2018)
M.D. Skowronski et al.
Automatic speech recognition using a predictive echo state network classifier
Neural Netw.
(2007)
M.H. Tong et al.
Learning grammatical structure with echo state networks
Neural Netw.
(2007)
S.X. Lun et al.
An I-V model based on time warp invariant echo state network for photovoltaic array with shaded solar cells
Solar Energy
(2014)
X.S. Yao et al.
Identification method for a class of periodic discrete-time dynamic nonlinear systems based on sinusoidal ESN
Neurocomputing
(2018)
G.K. Venayagamoorthy
Online design of an echo state network based wide area monitor for a multimachine power system
Neural Netw.
(2007)
J. Huang et al.
Echo state network based predictive control with particle swarm optimization for pneumatic muscle actuator
J. Frankl. Inst.
(2016)

L. Bozhkov et al.

Learning to decode human emotions with echo state networks

Neural Netw.

(2016)

S. Scardapane et al.

A decentralized training algorithm for echo state networks in distributed big data applications

Neural Netw.

(2016)

R. Fletcher et al.

A rapidly convergent descent method for minimization

Comput. J.

(1963)

P. Rebentrost et al.

Quantum gradient descent and Newton’s method for constrained polynomial optimization

(2016)

H. Jaeger et al.

Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless telecommunication

Science

(2004)

H. Jaeger

The ‘echo state’ approach to analysing and training recurrent neural networks–with an erratum note

GMD Report 148

(2010)

H. Jaeger

A tutorial on training recurrent neural networks, covering BPTT, RURL, EKF and the ‘Echo State Network’ Approach

Technical Report GMD Report 159

(2002)

H.G. Zhang et al.

A comprehensive review of stability analysis of continuous-time recurrent neural networks

IEEE Trans. Neural Netw. Learn. Syst.

(2014)

Cited by (38)

A novel broad learning system integrated with restricted Boltzmann machine and echo state network for time series forecasting
2024, Engineering Applications of Artificial Intelligence
Time series data prediction is crucial in system control, social management, and economic production. For the complex features of time series data and the massive amount of arithmetic in deep learning, a novel network model is proposed based on the broad learning architecture to handle time series prediction tasks. The model utilizes the restricted Boltzmann machine (RBM) in the mapping layer to learn feature information from the input data. Simultaneously, it employs the echo state network (ESN) as the fundamental unit in the enhancement layer to fit the learned feature information from the mapping layer. The proposed model's predictive performance has been validated on air quality index, PM2.5, and electric power datasets. Compared with the benchmark model, it has the advantage of training speed and can reduce prediction error by up to 36%, proving its effectiveness in time series prediction tasks.
Multi-Reservoir Echo State Networks with Hodrick–Prescott Filter for nonlinear time-series prediction
2023, Applied Soft Computing
The Echo State Network (ESN) is a representative model for reservoir computing, which is capable of high-speed model training for machine learning tasks with time series data. Extended models of the ESN, such as Multi-Reservoir ESNs (MRESNs), have been intensively studied for performance improvement in recent years. In this study, we propose a new model called an HP-MRESN by combining an MRESN with the Hodrick–Prescott (HP) filter for nonlinear time series prediction. The proposed HP-MRESN comprises three basic components: a time series decomposer, a reservoir state extractor, and an ensemble decoder. In the time series decomposer, we recursively leverage the HP filter to decompose original time-series data into multiple trend and cycle components. In the reservoir state extractor, each time series component is fed into a corresponding reservoir-state encoder for generating a reservoir state which is extracted as it is or through the principal component analysis. In the ensemble decoder, the states of multiple reservoirs are collected and processed to produce model outputs. Moreover, we propose a greedy algorithm to automatically find the best model architectures under designated hyperparameters for different prediction tasks. Experimental results on a total of 24 nonlinear time-series prediction tasks with 6 real-world datasets demonstrate that our proposed HP-MRESN not only can outperform some existing representative MRESN models and fully-trained RNN models but also can have relatively low training time. In addition, performance comparisons between the HP-MRESN and related MRESN models with other prepossessing methods show the benefit of time series decompositions using the HP filter. The codes of the proposed method are publicly available on https://github.com/Ziqiang-IRCN/HP-MRESN.
Error-output recurrent multi-layer Kernel Reservoir Network for electricity load time series forecasting
2023, Engineering Applications of Artificial Intelligence
Citation Excerpt :
At the same time, three types of classic randomization-based learning algorithms were employed to tackle the time series prediction problem and power systems. For example, ESN achieved good performance in time series prediction (Yao and Wang, 2019; Zheng et al., 2020). ELM played a vital role in short-term wind speed prediction (Wang et al., 2018; Mi et al., 2017).
Electricity is one of the most consumed commodities in the modern world. Electricity load prediction models are used to plan distribution operations to balance the equilibrium of demand and supply. This necessity has increased the number of recent research works. They employed several learning algorithms, such as support vector regression, to predict demands. However, these algorithms have high computational cost and too many user-defined parameters that directly impact their performance. Recently, randomization-based learning algorithms have been widely tested because they performed well at a lower cost. However, still, there was a main drawback: uncertainty in approximation and learning. This work employed a kernel trick to solve the uncertainty problem. A kernel with reservoir-state layers was used to solve the problem. The kernel reservoir-state layers from the echo state network not only transformed features into high-dimensional space, but also enhanced the forecasting ability by learning temporal information. Additionally, the proposed model also had a multi-step prediction ability that used previous forecasting errors to update the output weights in the current step to prevent an accumulated error problem. We compared our proposed model with single-layer and multi-layer variants of Extreme Learning Machine, Echo State Network, and Random Vector Functional Link on ten electrical load data sets. The proposed model showed the best performance on 9/10 data sets in terms of Mean Square Error or Symmetric Mean Absolute Percentage Error. These findings implied that the proposed algorithm was superior in forecasting long-term electricity load.
Echo state network with multiple delayed outputs for multiple delayed time series prediction
2022, Journal of the Franklin Institute
Citation Excerpt :
Case 2): For the NARMA-5 Compared with other methods (ESN [1], Leaky-ESN [9], RVESN [32], MAR-DESN [34] and Broad-ESN [33]), the prediction accuracy of these methods is shown in Table 6. The output curves of the MDO-ESN and ESN are shown in Fig. 5, and the error curves of the MDO-ESN and ESN are shown in Fig. 6.
In this paper, considering the influence of multiple delayed output items on the prediction accuracy of echo state network, a novel echo state network with multiple delayed outputs (MDO-ESN) is proposed for time series prediction with multiple delayed outputs. Firstly, for a given learning task, through studying the autocorrelation of output signal, its delayed characteristics can be determined, and then the corresponding delayed item of output equation of the MDO-ESN can be adjusted adaptively. Secondly, in order to improve the adaptability of the MDO-ESN in different learning tasks, a sufficient condition is given to satisfy the stability of the MDO-ESN. Thirdly, a parameter optimization method is given to reduce the dependence of the prediction accuracy of the MDO-ESN on the reservoir parameters of the MDO-ESN. Finally, two numerical simulation examples and one actual simulation example are used for verifying the effectiveness of the MDO-ESN.
Nonmonotone diagonally scaled limited-memory BFGS methods with application to compressive sensing based on a penalty model
2022, Applied Numerical Mathematics
According to a minimization problem founded upon the Byrd–Nocedal measure function coupled with a penalty term of the secant equation, a diagonal quasi–Newton updating formula is given. Then, the proposed update is applied as the initial (inverse) Hessian estimation of the limited-memory BFGS (Broyden–Fletcher–Goldfarb–Shanno) method. An improved version of the method is also given based on the Li–Fukushima modified secant equation. Convergence analysis for the general functions is conducted based on a nonmonotone Armijo line search. Eventually, practical advantages of the methods are computationally appraised through some CUTEr functions and also, on the compressive sensing problem.
Online robust echo state broad learning system
2021, Neurocomputing
Citation Excerpt :
The weights used by their state matrix are randomly generated, which makes them have better generalization ability while improving the learning speed. Several ESN-based methods have been proposed for multivariate time series classification through ingenious constructions of reservoirs[14,15]. In recent years, broad learning system (BLS) is proposed as an emerging and efficient randomized neural network, which is established on the flat network architecture.
Recurrent broad learning system (RBLS) is an effective learning method for processing sequential data. By replacing the enhancement nodes of the broad learning system with recurrent structure, RBLS obtains the capacity to capture the dynamic characteristics of time series data. However, RBLS is derived under the minimum mean square error (MMSE) criterion, which is sensitive to outliers. Moreover, RBLS is insufficient for online sequential learning. To address these limitations, we propose a novel online robust echo state structure based RBLS (OR-ESBLS). In OR-ESBLS, kernel recursive maximum correntropy (KRMC) is introduced to both enhance the robustness and discover the nonlinear characteristics of feature nodes in an online manner. To reduce the heavy computational requirements caused by the kernel method, a Quasi-Monte Carlo (QMC) based Random Fourier Feature (RFF) is utilized for kernel approximation. Furthermore, we adopt the randomized sparse reservoir as the enhancement nodes of RBLS, which can much more efficiently capture dynamic information of the data in the sequential learning setting. Experiments on both synthetic and real-world datasets are reported. The results show that the proposed OR-ESBLS can provide superior performance in online sequential time series prediction.

View all citing articles on Scopus

View full text

Broad echo state network for multivariate time series prediction

Abstract

Introduction

Section snippets

Broad echo state network

Learning algorithm of reservoir parameters and output weights

Simulation examples

Conclusion

Acknowledgment

Neural Netw.

Inf. Sci.

Neurocomputing

Appl. Soft Comput.

Neural Netw.

Neural Netw.

Solar Energy

Neurocomputing

Neural Netw.

J. Frankl. Inst.

Neural Netw.

Neural Netw.

Comput. J.

Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless telecommunication

Science

The ‘echo state’ approach to analysing and training recurrent neural networks–with an erratum note

GMD Report 148

A tutorial on training recurrent neural networks, covering BPTT, RURL, EKF and the ‘Echo State Network’ Approach

Technical Report GMD Report 159

A comprehensive review of stability analysis of continuous-time recurrent neural networks

IEEE Trans. Neural Netw. Learn. Syst.