Linear and nonlinear dynamic systems in financial time series prediction

Article history: Received April 28, 2012 Accepted 6 July 2012 Available online July 12 2012 Autoregressive moving average (ARMA) process and dynamic neural networks namely the nonlinear autoregressive moving average with exogenous inputs (NARX) are compared by evaluating their ability to predict financial time series; for instance the S&P500 returns. Two classes of ARMA are considered. The first one is the standard ARMA model which is a linear static system. The second one uses Kalman filter (KF) to estimate and predict ARMA coefficients. This model is a linear dynamic system. The forecasting ability of each system is evaluated by means of mean absolute error (MAE) and mean absolute deviation (MAD) statistics. Simulation results indicate that the ARMA-KF system performs better than the standard ARMA alone. Thus, introducing dynamics into the ARMA process improves the forecasting accuracy. In addition, the ARMA-KF outperformed the NARX. This result may suggest that the linear component found in the S&P500 return series is more dominant than the nonlinear part. In sum, we conclude that introducing dynamics into the ARMA process provides an effective system for S&P500 time series prediction. © 2012 Growing Science Ltd. All rights reserved.


Introduction
The modeling and forecasting of time series is an appealing yet difficult task in real world problems.Indeed, the prediction of time series is challenging and has received a large attention over the most recent decades in many applications such as power load (Zhou et al., 2012), traffic flow (Castro-Neto et al., 2009), weather forecast (Awan & Awais, 2011), sunspot prediction (Mirmomeni et al., 2010), electricity price forecasting (Niu et al., 2010) and stock market prediction (Liu et al., 2012); to name a few.A number of techniques to predict time series have been introduced in the literature including the well known autoregressive-integrated-movingaverage (ARIMA) processes and artificial neural networks (ANN).Introduced by Box and Jenkins (Box & Jenkins, 1976), ARIMA models originate from the autoregressive (AR) model, moving average (MA) model, and an integrated (I) part to make the data series stationary by differencing.The combination of AR and MA models gives the ARIMA process.However, ARIMA models assume a linear relationship between current and past values of a time series as well as with white noise.Indeed, they fit linear relations better than nonlinear relations.
In sum, the advantage of ARIMA is the fact that is it suitable to capture the stationary linear component, but it fails to model nonlinear structure.Therefore, approximations obtained by ARMA processes may not be adequate for complex nonlinear real-world data.In addition, ARIMA models are not capable to capture the dynamics of a time series.More precisely, the obtained AR and MA coefficients are static which make ARIMA a non dynamic system.On the other hand, ANN are nonlinear methods that can learn from patterns and capture hidden functional relationships in a given data even if the functional relationships are not known or difficult to identify (Zhang et al., 1998).In particular, they are capable of parallel processing of the information with no prior assumption about the model form.In particular, the ANN is determined by the characteristics of the data.In addition, ANN are adaptive; hence the network is capable to model non-stationary and dynamic data and systems.The ANN can be categorized into static and dynamic networks.
Static networks are feed-forward networks and do not have any delay or feedback elements, whilst the output of dynamic networks depends on the current, previous inputs or outputs of the network (Hagan et al., 1996).Although previous studies have reported that ARIMA models outperform ANN in terms of forecasting accuracy (Denton, 1995;Hann & Steurer, 1996;Taskaya & Casey, 2005) several recent papers in the literature have shown the superiority of ANN over ARIMA and that integration of ARIMA and ANN (for instance multi-layer back-propagation neural network) can be an effective way of improving upon their predictive performance (Zhang, 2003;Khashei & Bijari, 2010, 2011, 2012).ARIMA models were also combined with other machine learning techniques for time series prediction.For instance, Tseng et al. (2001) proposed a hybrid system called FARIMA that combines the fuzzy regression and ARIMA models to forecast foreign exchange market time series.Pai and Lin (2005) proposed a hybrid system based on ARIMA and support vector machines (SVM) for stock prices forecasting.Chen and Wang (2007) proposed hybrid model that incorporates seasonal autoregressive integrated moving average (SARIMA) model and support vector machines for seasonal time series forecasting, and Zhou and Hu (2008) proposed a hybrid modeling and forecasting approach based on grey and the Box-Jenkins autoregressive moving average models to predict gyro drift.In all previous papers, the effect of hybridization was significant on the improvement of the forecasting accuracy.
The purpose of this paper is twofold.First, it presents a hybrid model that combines ARIMA and Kalman filter (Kalman, 1960) to model and forecast time series.Since the Kalman filter is a well-known as an efficient recursive algorithm that estimates the state of dynamic systems, it used to update the ARIMA process coefficients.In other words, ARIMA is represented in a state space form.As a result, it is no longer static since its coefficients are dynamic and change over time.Unlike the standard ARIMA which is a static model, the main advantage of the ARIMA-Kalman model is to account for the dynamics of the time series and the possibility to improve the forecasting accuracy.Second, the nonlinear autoregressive moving average with exogenous inputs (NARX) which is a dynamic neural network suitable for modeling dynamic systems and time series prediction is used for comparison purpose.Indeed, the NARX has been successfully applied in modeling the dynamics of time series and complex systems (Menezes & Barreto, 2008;Arash & Atry, 2009;Pisoni et al., 2009;Pham et al., 2010).In particular, it is suitable for long-term prediction of univariate time series (Hagan et al., 1996).The performances of ARIMA-Kalman and NARX are compared in the context of financial time series prediction.For instance, the two models are applied to the problem of Standard&Poors (S&P500) returns forecasting.
The rest of the paper is organized as follows.In Section 2, ARIMA, Kalman filter and NARX are described.Experiments of the ARIMA, ARIMA-Kalman and NARX models are compared in Section 3. Finally, the conclusions are presented in Section 4.

The ARMA model
The ARMA(p,q) model represents the future value of a variable as a linear function of past observations and random errors.For instance, the ARMA(p,q) process of a stationary time series Z is given by: where t is time script, p is the order of the auto-regressive component, q is the order of the moving average component,  and  are coefficients to be estimated, and a are random errors which are assumed to be independently and identically distributed with a mean of zero and a constant variance  2 .The Box and Jenkins (1976) methodology includes three steps to identify the ARIMA parameters p and q.The autocorrelation function (ACF) and the partial autocorrelation function (PACF) of the sample data are used to identify the orders p and q.Finally, the equation ( 1) is estimated by maximum likelihood estimation.

The ARMA-KF model
The Kalman filter (KF) is an adaptive method of state variables and data modeling and estimation.The KF is the minimum variance estimation in linear systems within the realm of stochastic dynamic mode (Nasseri et al., 2011).The method recursively estimates the state variables of a system after measuring them.Thus, the KF is capable of making estimations of past, present and future states based on linearity and Gaussian assumptions of the underlying process.Consider the ARMA(p,q) process in equation ( 1) and let Then, the ARMA(p,q) process can be rewritten as: Thus, if the state space S t is defined by: Then, the observational equation is given by: and the state-transition equation that relates current and previous state variables is given by where, More details about state space representation of ARMA models by Kalman filter can be found in Harvey (1993).

The NARX neural network
The Nonlinear Auto-Regressive with Exogenous inputs (NARX) neural network (Leontaritis & Billings, 1985) is a class of discrete-time nonlinear systems that can represent a variety of nonlinear dynamic systems.In particular, the NARX network is a dynamic neural network that contains recurrent feedbacks from several layers of the network to the input layer.It can be mathematically represented as follows: where u(t) and y(t) are respectively the input and the output of the system at time t, d u 1, d y 1, d y d u , W is a weights matrix, and f is an unknown nonlinear function to be approximated by a multi-layer perceptron.As a result, the general NARX network equation can be written as: where f h and f o are the hidden and output functions, w ih , w jh and w ho are the weights, and b h and b o are biases.All data fed to NARX network are normalized in [-1,1] interval.

Simulation results
The data analyzed in this study include the daily return of the S&P500 price index from January 27 th , 2000 to January 31st 2008; a total of 2014 observations.Figure 1 exhibits the daily returns of the S&P500 calculated as first logarithmic differences.The data set is divided into two parts.The first group of the data (90% of observations) is used for training the model.The second group of the data (10% of observations) is used for testing.For evaluation purpose, two typical indicators the mean absolute errors (MAE) and the mean absolute deviation (MAD) are used to compare predicted and actual values to validate each forecasting system.The error can be estimated by MAE and MAD defined as: where R t , F t , and F are respectively the observed return, forecasted return and the average of forecasted return over the testing (out-of-sample) period; for example t =1 to N. The simulation results are shown in Figure 2. A clear comparison of the three methods (ARMA, ARMA-KF, NARX neural network) for forecasting of the S&P500 is reported via MAE and MAD.The obtained results indicate that the prediction performance of the ARMA-KF as a dynamic linear prediction system is better than the performance of the single ARMA time series forecasting model.This shows that introducing dynamics into the standard ARMA process is an effective and promising approach to improve its performance in S&P500 time series forecasting.
In addition, it is quite clear that the NARX neural network which is a nonlinear dynamic system performs better than the standard ARMA process.However, the NARX neural network underperforms the ARMA-KF model.At first sight, it may be surprising that the dynamic nonlinear model obtains worst results in terms of MAE and MAD than the linear model ARMA-KF.This finding could be explained by the fact that the nonlinear structure of the considered time series could be sufficiently weak so that the data generating process could be better approximated by a dynamic linear process such as the ARMA-KF.

Conclusion
In this study, ARMA process, ARMA coupled with Kalman filter (ARMA-KF), and the Nonlinear Auto-Regressive with Exogenous inputs (NARX) neural network have been compared for financial time series prediction.In terms of empirical results, we have found that across these predictive systems for the test case of the S&P500 variations and on the basis of two evaluation criteria namely the mean absolute errors (MAE) and mean absolute deviation (MAD), the ARMA-KF which is a linear dynamic predictive system performs the The findings demonstrate the predicting potential of the ARMA-KF system in financial applications.In particular, the results indicate that ARMA-KF can be a useful tool for S&P500 returns prediction.For future work, a combination of the ARMA-KF and NARX systems could be considered to check whether this hybrid system could improve the forecasting accuracy.Other market indices will be also considered.