Forecast of Chaotic Series in a Horizon Superior to the Inverse of the Maximum Lyapunov Exponent

In this article, two models of the forecast of time series obtained from the chaotic dynamic systems are presented: the Lorenz system, the manufacture system, and the volume of the Great Salt Lake of Utah. The theory of the nonlinear dynamic systems indicates the capacity of making good-quality predictions of series coming from dynamic systems with chaotic behavior up to a temporal horizon determined by the inverse of the major Lyapunov exponent. The analysis of the Fourier power spectrum and the calculation of the maximum Lyapunov exponent allow confirming the origin of the series from a chaotic dynamic system. The delay time and the global dimension are employed as parameters in the models of forecast of artificial neuronal networks (ANN) and support vector machine (SVM). This research demonstrates how forecast models built with ANN and SVM have the capacity of making forecasts of good quality, in a superior temporal horizon at the determined interval by the inverse of the maximum Lyapunov exponent or theoretical forecast frontier before deteriorating exponentially.


Introduction
The forecast of time series allows the decision-maker to formulate strategies [1].The qualities of the evaluated forecasts through the indicators of performance are essential in the atmospheric sciences and in manufacture, economy, and physical science [2][3][4].The limits of the forecasts of the chaotic time series are theoretically defined by the value of the inverse of the maximum Lyapunov exponent.This value establishes the maximum forecast steps of the time series.The research analyzes the forecasting capacity in the steps after the theoretical boundary using ANN and SVM.
The research addresses the chaotic time series with defined characteristics by phase spaces and their behavior associated with the initial conditions of the variables, and the studied time series in the research have an erratic or chaotic behavior.This type of series is analyzed through algorithms to define the stable or chaotic origin of the dynamic system, the generator of the series.The characterization of the dynamics of the system allows adjusting the different models of forecast of the time series [5].Takens's insert or encrust theorem is used to rebuild the dynamic system and its phase space from a particular set of data [5,6].Finally, this rebuilt system allows forecasting future periods through the information of the m-dimensional rebuilt dynamic system to forecast, in one or several steps, the time series.
For the chaotic time series, the forecast models will have an exponential divergence between the true values and the predicted ones.According to [7,8], the predictability time (T P ) of a system for a disturbance (ε) with an error (Δ) and with the maximum Lyapunov exponent (λ max ) is given by When modifying (1) for a time series, together with the elimination of errors and associated disturbances, (2) can be obtained; this equation defines the predictability time for a chaotic temporary series or forecast horizon (F H ).
The forecast models used are ANN and SVM, due to their capacity of recognizing patterns in databases.The strategies to predict vary depending on the dynamic system and the experiments made with the time series.The possible strategies are the combinations between single or multiple inputs and single or multiple outputs.
This research studies the forecast error and its later exponential growth to F H in chaotic temporary series, through the forecast models ANN and SVM.The studied series are stationary; its probability of change of state does not depend on time.
In [9,10], ANN is used based on the mobile average autoregressive to forecast in several steps coming from the chaotic attractor of Lorenz; it demonstrates an exponential decay of the capacity of forecasting in the time steps after the forecast frontier.The forecast of the series of Lorenz through ANN is studied, based on functions of radial basis and hybrid grouping methods to refine the parameters of the functions of radial basis.The forecast capacity with a determination coefficient of 0.9 up to 90 periods is maintained.
Authors propose the study of a series coming from a system based on a differential equation with the Mackley-Glass delay.This series is forecast with an ANN of 3 hidden layers, getting capacities of forecasting up to 120 steps.During the forecast competition NN3, the lazy method allows forecasting the series with the symmetric mean absolute percentage error (SMAPE) = 16.5%, using multiple entries and multiple outputs.
Authors such as [11][12][13] use support vector machine regression (SVMR) with different strategies, with the objective of forecasting different chaotic series.It is determined that SVMR with multiple exits presents better results on iterative machines.
According to [14], the chaotic system of Lorenz is used to validate SVM with minimum squares.The structures of Bayesian evidence have a better performance in relation to an SVM without this structure.These authors study a series corresponding to the process of unloading materials in a mining industry; in this research, the phase space of the system is rebuilt, getting from the forecast of the series a relative error in the rank 0.1145 and 0.0162.
In [15], a series of the price of coal is forecast through the ANN modifying the number of neurons in the hidden layer, resulting in an optimum of 7 neurons with a mean absolute percentage error (MAPE) = 1.79% in the stage of testing.On the other hand, authors such as [16] use an ANN with genetic algorithms to optimize the weights of the network; the objective is to forecast a series corresponding to the power of the wind, and the results show that this type of hybrid network is better than the normal ANN.
The study of [17] uses an ANN with restricted machines of Boltzmann, with an optimization of a colony of particles to decide the size of the network.Its objective is to forecast the series known as "the CATS benchmark." In [18], a time series coming from a chaotic circuit of a resonator diode is forecast, through a network of perceptrons with multiple layers in two stages.This network optimizes the combination of the prediction by nearby neighborhoods, getting a value of the mean squared error (MSE) equal to 0.114 with a prediction up to 20 steps.
On the other hand, authors such as [19] use the same concept of dynamic neighborhoods with local adaptive models to forecast tides under different conditions in the North Sea of Europe.Different chaotic uni-or multivariable models are made with direct prediction and multiple steps, with the objective of comparing them with ANN.The results of the forecast obtained by the different ANN models and the chaotic models of direct prediction do not show meaningful differences.
Authors such as [20] use SVM to predict the characteristics of the channel of a communication line with low voltage power; the results showed a decay of the capacity of the forecast when prediction steps increase.The bibliographic search does not find predictions beyond the frontiers of the forecast for different times of sampling.
On the other hand, in [21], an ANN with a hidden layer is used to predict iteratively up to F H the monitoring system of bridges, getting a MSE of 0.0185.Authors such as [22] propose a local model of polynomials of Chebyshev to predict the fall of the water into the river using polynomials of the second and third degree.The relative error in F H grows exponentially in relation to the previous periods.
Finally, authors such as [23,24] study the predictive capacity of the model of ANN to evaluate the value of copper in the international markets.They study nonlinear characteristics of the time series and determine its behavior through the reconstruction of the chaotic attractor and the analysis of visual recurrence.The results allow defining two cycles of the price of the copper and making very short-term forecast of the values of the price of copper always within F H of the maximum Lyapunov exponent.
All the researches establish the capacity of the forecast of the ANN and SVM methods, together with their performance indicators within F H . Based on the previous investigations, in this research, the performance of the indicators outside of theoric F H is studied, up to the limit defined by the exponential growth of the performance indicators.

Materials and Methods
2.1.Dynamic Characterization of the System.Takens's theorem is used to reproduce the geometric structure of the multidimensional dynamic or strange attractor y n , through a temporary measurement of a variable.Equation (3) corresponds to the point m-dimensional of the dynamic system with the time n, with n-umpteenth elements of the time series, delay time (τ), and global dimension (m).
According to [25,26], the elements obtained from a time series that characterized the dynamic system are as follows: (1) Periodicity: the spectrum of power studies the nature of a signal.It is calculated as the square of the absolute value of the discrete Fourier transform.For series with chaotic behavior, the spectrum will be continuous in the interval of frequencies.
(2) Delay time: the delay time defines the distance among elements of the series for the calculation of the global dimension.The most important delay time corresponds to the value τ where the first minimum of the function of the mutual average function is obtained.
(3) Global dimension: it is the necessary dimension to be able to reproduce the dynamic of the system.Calculated τ can use the algorithm of the next false neighborhoods and corresponds to the value where the percentage of false neighbors is zero.
(4) Maximum Lyapunov exponent: the distance of these points in a time n is given the two next points and is given by where δ 0 is the initial distance of the points, e is the Euler constant, λ is the Lyapunov exponent, and Δn is the time differential between near points.The system can have more than one exponent, but the greatest value to evaluate the existence of a chaotic system is studied.For a chaotic system, the value of the exponent is found in the rank 0, 1 .The Kantz algorithm is used for the calculation of the exponent.
2.2.Forecast Models.The application of the theory of nonlinear dynamic systems allows the adjustment of the forecast models.The entry of the models uses the coordinates m-dimensional of the attractor point in a time n.

Artificial Neuronal Networks.
A neuronal backpropagation network with only a layer of hidden neurons is used, and a training of Bayesian regularization is performed to adjust the weights and trends of the hidden layer and the output layer.This procedure of learning is based on the heuristic algorithm of nonlinear Levenberg-Marquardt optimization.
The activation function Z k is a hyperbolic tangent function for the neurons of the hidden layer and a linear identity function for those of the output layer.Thus, the output of the kth neuron is represented in where θ jk and θ ok are the weights and trends of the output layer, respectively, w ij and w oj are the weights and trends of the hidden layer, respectively, and x i are the elements of the time series.

Support Vector Machines.
According to [27], SVM use the model of minimum squares, with restrictions of equality, a function of quadratic cost and a function of nonlinear mapping φ over the entry vectors w T due to the chaotic characteristics of the series.This mapping allows the containment of the points in a superior space where they are linearly contained by hyperplanes y k .The problem of optimization in the primal space is defined: subject to where γ is a parameter, x k is the n-umpteenth element of the series, b is the trend, and e k is the containment error.Later, it is proceeded to transform the previous problem to its version in the dual space.Besides, the expansion of the mapping resulting in a kernel function (K) is made: used in this research is the function of radial basis, where σ is a parameter of Finally, in (9), the resulting approach of the optimization problem in the dual space y x is where α belongs to 0, γ .

Methodology
The methodology is defined by two steps: the dynamic characterization of the system and its simulation, forecast, and performance of the model.The dynamic characterization of the system is defined by the mathematical development of the spectrum of power, the mutual average information function, the algorithm of the near false neighbors, and finally the calculation of the maximum Lyapunov exponent.
It is defined as an experiment S j with j as the number of periods to forecast.Different experiments are proposed S j > F H , where F H is the number of steps of the frontier of the forecast.For each S j , the attractor is reconstructed and, with the information of the phase space, the model of the forecast with its matrices IN and OUT is built ((10)).Each matrix will have a total of N − m − 1 * τ − S j lines, where N is the number of elements in the series.3 Complexity In the process of simulation of the parameters of each model, the modification of the number of neurons in the hidden layer is made for each ANN.For the case of the SVM, γ parameters are varied of the problem of optimization and σ of the function of radial basis.The parameters of the SVM were settled, and the training of the model with 80% of the matrices IN and OUT is carried out, as well as the sampling with the remaining 20% of the matrices.In the stage of sampling for both forecast models ANN and SVM, the performance indicators are calculated.Finally, the parameter that has the minimum values of the indicators MSE, MAPE, RMSE, and index aggregation (IA) is chosen.According to [28,29], the Lorenz dynamic system is described by three nonlinear differential equations that describe a fluid in the function of position and time.To determine the effects of creating a strange attractor, the following parameters are used: σ L = 16, ρ = 45 92, and β = 4.With the objective of getting a series of 12001 elements, the Runge-Kutta method of the 4th order with a step of 0.01 is used.The series used is x t (12) (Figure 1).
A System of Flexible Manufacture.The model of flexible manufacture does not have a program of determined ordering; the assignment of operations to the machines is made according to the stage of the production system, which is the intrinsic characteristic of the system of flexible manufacture.The assembly cell is simulated, and a time series with 12001 data corresponding to the average number of pieces is obtained, in the waiting line of this cell (Figure 2).
3.1.3.Great Salt Lake of Utah, EEUU.Empirically, the Great Salt Lake of Utah is the oldest time series in the world.It corresponds to the elevation of the Great Salt Lake (GSL).The simple form corresponds to 4029 data.With the following polynomial transformation changes at the height of GSL (h) at the station Saltair Boat Harbor in the total volume of the lake, this transformation is valid for h belonging to 4170, 4215 .
The transformation of h to the series of the total volume of GSL is shown in Figure 3.

Results and Discussion
The spectrums of power are shown in Figure 4 for the series x t , Figure 5 for the series of the average number of pieces in the waiting line, and Figure 6 for the total volume of GSL.The τ optimum, m, λ max , and F H are summarized in Table 1.
4.1.Series of the Lorenz Case.For the forecast of the series x t with ANN, the parameter of neurons of the hidden layer for each one of the experiments S j is shown.This parameter corresponds to the value with the minimum average values of the indicators MSE, MAPE, and RMSE and the maximum average value of IA.The indicators have the minimum values for all the experiments for the first forecast period and the maximum value in the last of the periods.The indicator IA behaves in a contrary way, and its greatest value is for S j = 1 (Table 2).
The evolution of the indicators in the different forecast periods is shown in Figure 7.The lines of the graphic are colored according to the number of steps after F H : with solid lines in yellow for S j = 12, black for S j = 6, green for S j = 4, red for S j = 3, and blue for S j = 2.
The forecast of the series x t through SVM shows the parameters chosen for each S j .The parameters chosen represent the best performance of the forecast model with the evaluation of the indicators MSE, MAPE, RMSE, and IA.
The evolution of the indicators is represented through the length of the forecast periods: dashed lines in yellow for S j = 12, black for S j = 6, green for S j = 4, red for S j = 3, and blue for S j = 2.The behaviors of the characteristics of the experiment with SVM in the twelve periods were studied (Figure 8 and Table 3).

Series of the Average Number of Pieces in the Waiting
Line Case.Table 4 shows the forecasts of the series of the average number of pieces in waiting line through ANN; in this table, the parameters chosen for neurons in the hidden layer for each one of the experiments S j are observed.The parameter corresponds to the lower average value of the indicators MSE, MAPE, and RMSE and the maximum value of the IA.All the indicators maintain the same decreasing behavior for MSE, MAPE, and RMSE.The only exception corresponds to the indicator MAPE with a minimum value in the step S j = 4.
Finally, the evolutions of the indicators through the forecast periods are presented: solid lines for ANN and dashed lines for SVM.The colors represent the steps for each method: yellow for S j = 12, black for S j = 6, green for S j = 4, red for S j = 3, and blue for S j = 2 (Figure 9).4.3.Total Volume of the Great Salt Lake Case.The results of the forecast of the series of the total volume of GSL, through ANN and SVM, are presented in Figure 10, with a maximum   7 Complexity step after the theoric frontier of forecast S j = 20.Figure 10 presents the evolutions of the indicators through the forecast periods: solid lines for ANN and dashed lines for SVM with steps S j = 20 (purple), S j = 15 (orange), S j = 10 (black), S j = 8 (green), S j = 6 (red), S j = 5 (blue), and S j = 3 (yellow).

Conclusions
For the three time series studied, both models have capacities to forecast beyond the theoric frontier before deteriorating exponentially.The main results of the investigation were as follows: (1) The forecast of the series of Lorenz develops exponential errors in period 6 with the SVM method and in period 10 with the ANN method.This series with the SVM forecast model has better performance indicators for the period forecast in all the experiments.
(2) The series of the manufacture system and the GSL series required 12 and 20 periods after the theoric limit, to obtain the exponential deterioration of the performance indicators through the ANN method.
(3) The forecast of the series of the system of flexible manufacture and GSL has a better performance with the indicator of squared minimums, for most of the periods after the theoric limit in all the experiments, using the method ANN backpropagation in relation to the SVM method.
(4) This study allows forecasting those series beyond theoric F H , without an exponential deterioration of the performance indicators.
(5) The improvement of forecasts was made using the ANN and SVM methods adjusted with the parameters determined by the chaotic characteristics of each series.Great Salt Lake h:

Notations
Height of the Great Salt Lake.

Figure 1 :
Figure 1: Variable x t of the Lorenz system.

Figure 2 :
Figure 2: Average of parts in the waiting line.

Figure 3 :
Figure 3: Volume of the Great Salt Lake (acre * feet).

Table 1 :
Characteristics of the series.

Table 3 :
Parameters for the manufacturing system.

Table 4 :
Parameters for the Great Salt Lake.
jk and θ ok : Weights and trends of the output layer w ij and w oj : Weights and trends of the hidden layer