Feed Forward Neural Network Modeling for Rainfall Prediction

. Prediction of rainfall data by using Feed Forward Neural Network (FFNN) model is proposed. FFNN is a class of neural network which has three layers for processing. In time series prediction, including in case of rainfall data, the input layer is the past values of the same series up to certain lag and the output layer is the current value. Beside a few lagged times, the seasonal pattern also considered as an important aspect of choosing the potential input. The autocorrelation function and partial autocorrelation function patterns are used as aid of selecting the input. In the second layer called hidden layer, the logistic sigmoid is used as activation function because of the monotonic and differentiable. Processing is done by the weighted summing of the input variables and transfer process in the hidden layer. Backpropagation algorithm is applied in the training process. Some gradient based optimization methods are used to obtain the connection weights of FFNN model. The prediction is the output resulting of the process in the last layer. In each optimization method, the looping process is performed several times in order to get the most suitable result in various composition of separating data. The best one is chosen by the least mean square error (MSE) criteria. The least of in-sample and out-sample predictions from the repeating results been the base of choosing the best optimization method. In this study, the model is applied in the ten-daily rainfall data of ZOM 136 Cokrotulung Klaten . Simulation results give a consecution that the more complex architecture is not guarantee the better prediction.


INTRODUCTION
Neural network modeling for rainfall prediction has been rapidly developed in recent many years.It desiderated can respond the need of reliable modelling in rainfall prediction.Understanding the pattern of rainfall data and get an accurate prediction are very important tasks in hydrology and climate fields.Reliable rainfall prediction also give a great impact in forecasting the rainfall data on daily, ten-daily or monthly and seasonal time scales.It provide useful information for water resource management, agricultural planning, and associated crop insurance application significant implications for food production, securing water supplies for major population centres, and minimizing flood risks.Some researches about rainfall prediction by using neural network have been conducted conducted (Cigizoglu et al, 2009;Benmahdjouba et al., 2013;Asadi et al., 2013) [1][2][3].Some other publications related to comparison and developing new models also have been conducted (Lin and Wu, 2009;Mekanik et al, 2013 ;Kim et al., 2016) [4][5][6].In this research we applied Feed Forward Neural Network (FFNN), a main class of neural network modelling, for rainfall prediction in ZOM 136 Cokrotulung Klaten.Some of the interesting parts of neural network modeling are procedures of determining the optimal input, the number of hidden unit, the activation function used in the hidden layer and the choosing of optimization method for obtaining the weights of the network.Various procedures have been evolved for obtaining the most efficient and reasonable network in both theoretically and practically.The optimization technique for obtaining network weights is one of the main focus in neural network modelling.Various standard gradient based methods have been used to estimate the weights.Therefore, the activation function permitted in the hidden layer should be continue and differentiable functions.In this research, the logistic sigmoid and the linear function are used as activation functions in hidden layer and output layer, respectively.Various architecture have been conducted for choosing the best one.In each architecture, the process is repeated several times for getting the least Mean Square Error (MSE) as the based of choosing the best architecture.The three gradient based optimization are used to optimize the neural network weights.

MATERIAL AND METHODS
In this section, the neural network model for time series prediction is explained.Neural network architecture consists of three layers namely input layer, hidden layer and output layer.Each layer has a connection with the previous layer and called weight model for determining the input.While, the overfitting and underfitting technique also often be used as inquiring the alternative ones.Some other researches utilize certain algorithm so that neural network model is free-standing from other model.In this research the best ARIMA model is chosen for determining input.Architecture of neural network for time series prediction is described in Fig. 1.Output layer just have one neuron because of the univariate case.If the network is applied in multivariate time series, the more than one neuron is needed for describing the model.

Figure 1. Neural network architecture for time series prediction
In the form of mathematical model, the network architecture can be described as follows. ( where, is activation function at hidden layer and the weights vector of the network consists of weights from input layer to hidden layer (wij), weights from bias to hidden layer (wbj), weights from hidden layer to output (wj), and weights from bias to output (wb).Therefore, the three optimization methods often used in neural network modeling for time series prediction is briefly discussed.
Backpropagation algorithm in the FFNN model consists of three stages, namely feedforward of the input pattern, calculating the error and adjusting the weights.In this algorithm, the initial weights are specified first.By the feedforward stage, the output of the initial points is calculated.Furthermore, the difference between output and target, namely error, is found.Weights updating is done by using a gradient based method for obtaining the smaller error.The termination criteria are specified for stopping the iteration process.Maximum epochs permitted and minimum error reached are the two basic rules commonly used as the stopping criteria.In this research we used the three gradient based methods for obtaining the weights of FFNN, that are Gradient Descent, Levenberg-Marquardt and Quasi Newton BFGS.Levenberg-Marquardt algorithm is a virtual standard in nonlinear optimization.It is the most widely used optimization algorithm.It outperforms simple gradient descent and other conjugate gradient methods in a wide variety of problems.In many cases, it finds a solution even if it starts very far off the final minimum.It is a very popular curve-fitting algorithm used in many software applications for solving generic curve-fitting problems.However, as for many fitting algorithms, it finds only a local minimum, which is not necessarily the global minimum.The Broyden-Fletcher-Goldfarb-Shanno (BFGS) method searches a stationary point of a (preferably twice continuously differentiable) function.A necessary condition for optimality of such problems is that the gradient be zero.If the function has not a quadratic Taylor expansion near an optimum, the BFGS method is not guaranteed to converge.This method uses both the first and second derivatives of the function.Gradient descent is a first-order optimization algorithm.To find a local minimum of a function, it need to takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point.Momentum method is an extension of Gradient descent method, that reduces the risk of a getting stuck in a local minimum, as well as speeds up the convergence considerably in cases where the process would otherwise zigzag heavily.Momentum simply adds a fraction of the previous weight update to the current one.When the gradient keeps pointing in the same direction, this will increase the size of the steps taken towards the minimum.It is therefore often necessary to reduce the global learning rate when we use a lot of momentum.

RESULTS AND DISCUSSION
Procedure of neural network modelling by using standard gradient based optimization methods is applied in the ten-daily rainfall data of ZOM 136 Cokrotulung, Klaten, Central Java Indonesia from January 2010 until July 2018 with the length of 309.We tried to divide the data into two parts with various ways.In the first experiment, the first 216 data (70%) is used as training and the remaining 93 data (30%) as testing.In the second experiment, the first 247 data (80%) is used as training and the remaining 62 data (20%) as testing, and the last experiment the first 278 data (90%) is used as training and the remaining 31 data (10%) as testing.In each composition, the same architecture is repeated for 30 times with 1-10 hidden units, respectively.The activation function used in hidden layer is logistic sigmoid.The maximum number of epochs in each experiment is 1000.The minimum root mean square error of each architecture is chosen for obtaining the best one.We need to refer to the values of the series in previous periods as the input.Determining lagged variables as input is performed by ARIMA model.In early step, the investigations resulting the lagged variables of 1, 2 and 18 as input.The summarize of the processes is shown in Table 1 whereas plot of the insample prediction of training data and plot of the outsample prediction of testing data by using the FFNN model in one of the iterations in the composition of 80:20 are shown in Fig. 1 and Fig. 2.  The results of Table 1 show that a better insample prediction is not give a guarantee that the outsample also better.In consequence, we should make decision of fitting the best one by using the two criteria,

Fig. 1
Fig. 1 shows the actual versus in-sample prediction of FFNN model in the rainfall data of ZOM 136.Generally, the seasonal pattern of the in-sample data can be followed by the prediction.The period of repetitive phenomena between both data are the same.None the worse, the model has a blind side.There are still several points far from as proper.Some points that cannot be reached by the model are the extreme values.It needs more extensive discussion for using the more suitable optimization methods for optimizing neural network related to the data type.The further open question also can be discussed about the more appropriate model for the data type.Similarly, the graph of testing data versus out-sample prediction in Fig. 2 also shows the resemblance between the two.Actual data which has an extreme value cannot be reached by the prediction but in general the out-sample prediction can keep abreast the pattern of actual data.
both in-sample and out-sample predictions.For the first composition, the 70% as training and 30% as testing, it is reasonable if we choose architecture with ten neurons in hidden layer and BFGS as optimization technique for constructing the selected one.The in-sample prediction has minimum of mean square error, while the out-sample prediction give a near the least one.Similar reason is considerable for choosing architecture which has nine neurons and BFGS optimization as the best architecture of the second composition.In the third separation, 90% as training and 10% as testing, selecting the optimal number of hidden units is debatable.The one choice is two hidden neurons -it is more balance between insample and out-sample-and the other one is nine hidden neurons.Both of them use Levenberg Marquardt as optimization method.The decision is depended on various factors.If the simplicity, parsimony and balanced results take into consideration, the first choice is better.The second choice is good for considering the strong capability in-sample prediction.4.CONCLUSIONProcedure of rainfall prediction by utilizing FeedForward Neural Network has been developed.The more complex network architecture not give a guarantee of a better prediction result.Levenberg Marquardt and BFGS algorithm can be regarded as the main optimization methods whereas Gradient Descent is not recommended for the data type.Comparison of the results with the other technique like heuristic optimization can be regarded as an interesting problem for the future work.

Table 1 .
MSE of the simulation results of the FFNN modelling for rainfall data in ZOM 136