COMPARATIVE ANALYSIS OF LSTM, RF AND SVM ARCHITECTURES FOR PREDICTING WIND NATURE FOR SMART CITY PLANNING

: Meteorological data and its effect have been the attention of the researchers of the smart city planning for thorough utilization and management of resources, that help in effective government management, convenient public services and sustainable industrial development. Renewable sources of energy like wind, solar, are being integrated into city planning to improve environmental quality. Wind energy is utilized through wind turbines and requires foreknowledge of wind parameters like speed and direction. The aim of this paper is to predict dominant wind speed and direction for time-series wind dataset, that can be incorporated into city planning for selecting suitable sites for wind turbines. This paper proposes three one-dimensional (1D) algorithms using Long Short Term Memory (LSTM), Random Forest (RF) and Support Vector Machine (SVM) for dominant wind speed and direction prediction. The proposed 1D LSTM (1DLSTM), RF (1DRF) and SVM (1DSVM) take successive time values in terms of wind speed and direction as input and predict the future dominant speed and direction, separately. The proposed algorithms are trained and tested using historical wind dataset of Stuttgart and Netherlands respectively. Prediction using 1DLSTM results in total accuracies reaching up to 93.9% and 94.7%, up to 92.8% and 93.8% using 1DSVM and up to 88.7% and 89.3% using 1DRF for speed and direction, respectively. Thus, prediction of wind nature using the proposed algorithms, will give city planners advanced knowledge of wind conditions.


INTRODUCTION
Due to the development and population growth, the energy demand is increasing.Daily increases in gas and oil prices, accompanied by the release of harmful chemicals from the combustion of these fossil fuels affects human health, environment, making human life miserable.Therefore, renewable sources of energy such as solar, wind, hydro have become popular as they provide a safe, clean and renewable environment.The uncertain nature of the wind makes it more demanding to model and predict its nature.The requirements of the installation sites for wind turbines would depend heavily on the impact of wind speed and direction (Reed et al., 2011, Aissou et al., 2015).Factors such as past wind speed and direction, temperature, pressure are responsible for the future wind trend.The installation of new wind turbines and sensors on site requires a future forecast and pre-evaluation of the wind nature accompanied by its highly governing wind speed and direction factors.The Numerical Weather Prediction (NWP) models (Louka et al., 2008), Machine Learning (ML) (Filik and Filik, 2017) and the combination of both NWP and ML (Vladislavleva et al., 2013) are used to model the wind prediction.The NWP approach is based on the physical kinematic equations, which use multiple meteorological variables as input for the prediction model and work by solving the complex mathematical models (Zhou et al., 2011).In ML various concepts can be used such as fuzzy logic (Monfared et al., 2009), neural networks (El-Fouly and El-Saadany, 2008, Daraeepour and Echeverri, 2014, Yesilbodak et al., 2017) and statistical models (Miranda and Dunn, 2006, Jursa and Rohrig, 2008, Zhou et al., 2011).Regression models using neural networks along with techniques like particle swarm optimization, wavelet transform (Martnez-Arellano et al., 2014), REP tree, M5P tree, bagging tree (Kusiak et al., 2009b, Kusiak andZhang, 2010), K-nearest neighbor algorithm (Jursa andRohrig, 2008, Treiber et al., 2016), principal component analysis, moving average models (De Giorgi et al., 2009, Vargas et al., 2010), Markov chain (Kusiak et al., 2009a, Treiber et al., 2016) have been used for wind analysis.Support Vector Machines (SVM) and its variation, Least Square Support Vector Machines (LSSVM) have also been used for forecasting wind speed (De Giorgi et al., 2009, De Giorgi et al., 2014).Lot of work has been done using ML algorithms for different types of wind predictions, very short term (some seconds to less than 30 minutes ahead), short term (30 minutes to less than 6 hours ahead), medium term (6 hour to less than 1 day ahead) and long term (1 day to less than 1 week ahead) time scales (Yesilbudak et al., 2013, Yesilbodak et al., 2017).Although (Shi et al., 2010) proposed a genetic algorithm-piecewise support vector machine model for short term wind power prediction.(Pourmousavi Kani and Ardehali, 2011) derived very short-term wind speed prediction using an artificial neural network-Markov chain model.(Yuan et al., 2015) proposed short-term wind power prediction based on the LSSVM-GSA model.Moreover recently (Kang et al., 2017) proposed short-term wind speed prediction using the combination of Empirical Mode Decomposition (EMD) with LSSVM, and hybrid EEMD i.e., EEMD-LSSVM Model.(Ghaderi et al., 2017) presented a spatio-temporal wind speed short term forecasting using LSTM taking wind mills in the northeast of the U.S. as a case study.(Jiang et al., 2017) used a spatiotemporal pattern network for predicting energy consumption giving a data-driven technique for producer and consumer side energy system.(Yusof and Zurita-Milla, 2017) mapped the frequent spatio-temporal wind profile patterns using multi-dimensional sequential pattern mining.Smoothening and filtering techniques used in several above discussed studies, ignore the noise in the data, and modify the original wind dataset.Above research suggests that there is, still a number of questions such as wind prediction for user desired time frame (long/short), without modifying the authenticity of the original temporal wind dataset, remain to be addressed.Thus, prediction methods which work on the original wind data by taking into consideration the noise, are still required.Therefore the current study proposes three one-dimensional (1D) algorithms based on LSTM, Random Forest (RF) and SVM with following contributions, (i) dominant wind speed and direction prediction using 1DLSTM, 1DRF, and 1DSVM, without applying any smoothening and noise removal techniques, (ii) the time-frame of prediction is user-defined, (iii) using 1D LSTM (1DLSTM), RF (1DRF) and SVM (1DSVM) as classification instead of regression to enhance accuracy, and (iv) comparative study of the 1DLSTM, 1DRF and 1DSVM architectures.The proposed models will provide foreknowledge of wind nature of an area, thereby helping in the proper selection of sites for wind turbine installation.This will provide more utilisation of renewable energy for safe and better city planning, that in turn would help in efficient management and development of the city's resources.The remaining paper is organized as follows, proposed methods and datasets employed are discussed in section 2 and section 3, respectively, section 4 explains the results, followed by conclusion in section 5.

METHODOLOGY
The wind dataset comprises of wind speed and direction with temporal resolution t and ti (i → 1 to k) denotes speed and direction at time i, where 1 and k are the first and last values in the dataset, respectively.Multiple samples are designed using the dataset for training and testing the proposed algorithms.A sample consists of a feature vector as an input with a corresponding output class.V b (a scalar) consecutive values of wind speed from ti to ti+V b form a feature vector of dimension V b × 1 which is the input of the sample.V f (a scalar) successive values of wind speed after the last value in the input i.e., ti+V b , are used to define the samples output class.Mean (µ), and standard deviation (σ) of the wind speed of the entire dataset are calculated.Various class boundaries are designed using µ and σ as shown in Table 1.Among V f , count of values occurring in each class in Table 1 is noted, and the class that has a maximum count i.e., dominant, is assigned to the sample.Similarly, multiple samples based on wind speed are created by taking V b values in the corresponding input from ti to ti+V b by varying i from 1 to k -V f , at an increment of 1.The outputs of these samples are designed as discussed above.Likewise, samples based on wind direction are created where direction instead of speed is considered both in the input and output and µ and σ of direction are calculated.Thus, at this stage, for V b values in the input from ti to ti+V b , there will be two sets of samples, one based on wind speed and other based on wind direction.
The proposed 1DLSTM is a special kind of RNN (Recurrent Neural Networks) capable of learning long term dependencies with chain-like structure.It has an input layer, four neural layers (N1, N2, N3, N4), i.e., three sigmoid layers along with tanh layer and an output layer.The input layer is 1D of the size of V b .The input layer is successively followed by 1D N1, N2, N3 and N4, with the output layer in the end.The output layer is a softmax layer (Goodfellow et al., 2016, Memisevic et al., 2010), having the number of neurons same as the number of the classes.There are seven classes in the present study as shown in Table 1.
The proposed 1DSVM algorithm classifies the data by finding the best hyperplane that separates all data points of one class from those of the other class.The best hyperplane for the 1DSVM signifies the one with the largest margin between the two classes.
The margin defines the maximal width of the slab parallel to the hyperplane that has no interior data point in time.The support vectors are the data point that are closest to the separating hyperplane; these data points are on the bounds of the slab.1DSVM can be used when data has exactly two classes.However, multiple classes can be classified using the one-vs-all (OVA) approach, one-vs-one (OVO), and all-vs-all (AVA) approach.In this study, OVO method along with nonlinear Radial Basis Function (RBF) kernel, have been used for classification.
The proposed 1DRF algorithm uses a decision tree as a decision support tool for classification.1DRF uses a tree-like graph to show the possible consequences.When the 1DRF is given a training sample, it formulates a set of rules which are used to perform predictions.Moreover, 1DRF uses sufficient decision trees, to ensure the classifier does not overfit the model while taking the average of all the predictions to remove the biases.The advantage of the 1DRF as a classifier is that it can handle missing values, and the classifier can be modeled for categorical values and to get the relative feature importance, that contributes in selection of the most favorable features for the classifier.Therefore 1DLSTM, 1DSVM and 1DRF are used to predict wind speed and direction separately.When predicting dominant speed, samples based on speed are used to train and test the 1DSVM.When the dominant direction is to be predicted, then the samples based on direction are used to train and test the 1DSVM.Similarly, the 1DLSTM and 1DRF are trained and tested with samples based on speed for dominant speed prediction and direction for dominant direction prediction.During training, the sample's feature vector of dimension V b × 1, forms the input of the 1DLSTM, 1DSVM and 1DRF, while the sample's output class forms the output of the 1DLSTM, 1DSVM and 1DRF.

DATASET
Stuttgart and Netherlands wind datasets are used in this study.In the corner of Hauptstaetter Strasse 70173 Stuttgart, the historical data from 30 years from 1987 to 2017 are taken from Stuttgart station sensor1 .This dataset contains the wind speed and direction with temporal information attached in a 30-minute interval.
The second dataset is of station 210 Valkenburg sensor from the Netherlands with 37 years of historical data from 1981 to 20182 .Each area's dataset is organised separately into individual month by using time information, with past data first, followed by current data.It helps to predict on a monthly basis, the dominant wind speed and direction.

RESULTS
The proposed algorithms were implemented using Python and executed with four cores on Intel R Core TM i7-4770 CPU @3.40 GHz.Stuttgart's 30-year historical data was separated by month to create monthly data.Similarly, according to each month, 37 years of historical data from the Netherlands was grouped.For each month data (individually for Stuttgart and Netherlands), each algorithm was executed separately to predict the dominant wind speed and direction.Several samples were created from the data of a month, each with input and corresponding output, as described in section 2.  Values of k1, k2 and k3 (Table 1) were determined empirically as 0.15, 0.45 and 0.65 for both speed and direction.It ensured that adequate number of samples was present in each class.Also, Synthetic Minority Oversampling Technique (SMOTE) was utilized to do up-sampling of the classes having inadequate number of samples.V b and V f were taken as 50 in this study.All the samples for a month, were randomly separated into training and testing samples with 40% of the total samples as testing samples.
The three proposed algorithms were trained and tested with these samples.In order to calculate the average accuracy values, the previous procedure of random division of the total samples into training and testing and the training of the proposed methods was repeated ten times, taking into account the randomness of division into training and testing.For Stuttgart data, minimum and maximum total accuracy for predicting dominant speed are 77.2% and 87.3% respectively using the proposed 1DRF method, 83.9% and 90.9% respectively using the proposed 1DSVM method, and 84.5% and 92.4% respectively using the proposed 1DLSTM method.Prediction of dominant direction of Stuttgart data, using the 1DRF results in minimum and maximum total accuracy of 81.6% and 89.3% respectively, 84.6% and 93.8% respectively using the 1DSVM and 87.4% and 93.3% respectively using the 1DLSTM (Figure 2 and Figure 3).Similarly, for Netherlands data, minimum and maximum total accuracy for predicting dominant speed are 80.7% and 88.7% respectively using the 1DRF method, while, 86.9% and 92.8%, respectively using the proposed 1DSVM method, and 87.7% and 93.9%, respectively, using the proposed 1DLSTM method. of dominant direction using the 1DRF method results in minimum and maximum total accuracy of 80.6% and 88.3% respectively, whereas 87.7% and 92.4%, respectively using the 1DSVM method and 87.2% and 94.7%, respectively using the 1DLSTM method, for the same Netherlands data (Figure 4 and Figure 5).Similar results were obtained for other months as well.Maximum precision and recall values are 92.6% and 92.7% respectively using 1DRF, 92.4% and 93.1% respectively using 1DSVM and 93.7% and 93.4% respectively using 1DLSTM.Further, value of V b was varied to carry out its sensitivity analysis.In first case, V b were varied and V f remains same as V b .In second case, V f was kept constant at 50 while V b varies.These were performed for June month of Stuttgart and total accuracy for dominant speed

Discussion
The proposed 1DRF using multiple decision trees is able to detect patterns in the input feature vector of a sample and is able to predict dominant wind speed and direction with good accuracy.The 1DSVM method maximizes the margin between the support vectors and the hyperplane and is able to perform better using a non linear radial basis function (RBF) kernel.The 1DSVM performs better by up to 8.4% and 6.4% in comparison to 1DRF for predicting dominant wind speed and direction respectively.The 1DLSTM with four recurrent layers with long term dependencies performs better by up to 9.3% and 7.9% in comparison to 1DRF for predicting dominant wind speed and direction respectively, and by up to 1.0% and 1.5% comparison to the 1DSVM.Moreover 1DLSTM and 1DSVM share simialr results for most of the dataset cases.The performance of 1DLSTM improved with increased numbers of neural layers with advanced activation functions, moreover it requires better hardware resources The higher number of classes ( 7) in the output ensures that the proposed methods are able to learn varieties of samples during the training and can predict with good accuracies during the testing.Moreover, the higher number of classes helps to identify the sudden changes in the wind speed and direction and ensures that most of the minor and major details are learnt during the training phase.During the designing of the samples, their output classes were decided statistically using µ and σ of a particular month's wind dataset, thereby representing the dataset better.However, currently with 7 classes in the 1DLSTM, 1DRF and 1DSVM, for Stuttgart, total accuracy enhances by up to 6.7% and 6.9% (for May month) using 1DLSTM and 6.3% and 6.6% (for May month) using 1DSVM for wind speed and direction, respectively, with respect to the corresponding month using 1DRF (Figure 2 and Figure 3).Similarly, for Netherlands, using the 1DLSTM, total accuracy is enhanced by up to 7.1% and 5.4% (for March month) and using 1DSVM by 6.9% and 5.5% (for October month) for wind speed and direction, respectively, with reference to the corresponding month using 1DRF (Figure 4 and Figure 5).The input feature vectors of samples are based on the original wind data values.The proposed 1DLSTM, 1DRF and 1DSVM algorithms take as input the original data without applying any smoothening technique to filter out the noise and have only a single user defined parameter V b , thus making these algorithms less susceptible to the noise along with the use of real data and minimum parameter tuning.The comparative study of the proposed 1DSVM with (De Giorgi et al., 2014) and (Yuan et al., 2015) is performed, along with the comparison of 1DLSTM with (Ghaderi et al., 2017), as these algorithms are nearest to the proposed methods.(De Giorgi et al., 2014) and (Yuan et al., 2015) have used SVM and LSTM but with regression analysis (LSSVM) and smoothening and filtering techniques have been applied to remove noise from the dataset.Similarly, (Ghaderi et al., 2017) has used LSTM with regression and noise has been removed from the dataset by smoothening and filtering, thereby modifying the originality of wind dataset.
The samples used in present study are utilized to train and test the (De Giorgi et al., 2014), (Yuan et al., 2015), and (Ghaderi et al., 2017) discussed architectures.In this case outputs of the samples are changed to real-values (i.e., regression) and unlike classification as in the 1DLSTM, 1DSVM and 1DRF.Values of V b and V f are kept as same.Symmetric Mean Absolute Percentage Error (SMAPE) (Shuyang et al., 2017) for wind speed using (De Giorgi et al., 2014), (Yuan et al., 2015), and (Ghaderi et al., 2017) are more or less similar and is 18.2% for V f = 15 and increases as V f increases, reaching up to 32.5% for V f = 50.Likewise results using (De Giorgi et al., 2014), (Yuan et al., 2015), and (Ghaderi et al., 2017) architectures were obtained for the wind direction.Thus, error increases substantially when more values are predicted in future using state-of-the-art SVM and LSTM based regression architectures (De Giorgi et al., 2014), (Yuan et al., 2015), and (Ghaderi et al., 2017).However, the proposed 1DSVM method for predicting dominant speed and direction based on classification, achieves high accuracy reaching up to 93.9% and 94.7% (1DLSTM), up to 92.8% and 93.8% (1DSVM) and up to 88.7% and 89.3% (1DRF) for speed and direction, respectively for V f = 50 even without applying any smoothening or filtering to the original data.Thus, the proposed 1DLSTM, 1DSVM and 1DRF methods are suited for predicting dominant speed and direction for a larger time period in the future unlike the (De Giorgi et al., 2014), (Yuan et al., 2015), and (Ghaderi et al., 2017) regression based architectures.

CONCLUSION
Meteorological data have been the attention of the researchers of the smart city planning for thorough utilization and management of resources, which help in effective government management, convenient public services and sustainable industrial development.Using renewable energy supply would provide a healthy and amiable city, and increased welfare in more general terms.
To ensure incorporation into the planning process, the renovation of the existing planning is indeed the most promising field for climate-related intervention.From a designers perspective, the authors have stressed the need to include energy-conscious strategies to improve environmental quality.The integration of new knowledge, innovative technologies in sustainable transformation is the motive of this paper.The algorithms using 1DLSTM, 1DRF and 1DSVM have been proposed for predicting the dominant wind speed and direction classes.V b continuous values of the wind speed and direction separately form an input sample and predict the dominating speed and direction, among V f values after the last value in the sample input, using 1DLSTM, 1DRF and 1DSVM.The proposed algorithms show promising results when trained and tested using wind datasets of Stuttgart and Netherlands.The maximum total accuracy using the 1DRF in case of Stuttgart for predicting dominant speed and direction are 83.7%,89.3% respectively and for Netherlands 88.7%, 88.3% respectively.Meanwhile, using the 1DSVM maximum total accuracy for predicting the dominant speed and direction for Stuttgart are 90.9%,93.8% respectively, and for Netherlands 92.8%, 92.4% respectively.Further, using the 1DLSTM maximum total accuracy for predicting the dominant speed and direction for Stuttgart are 92.7%,93.5% respectively, and for Netherlands 93.9%, 94.7% respectively.The total accuracy enhances by up to 6.7% and 6.9% (for May month) using 1DLSTM and 6.3% and 6.6% (for May month) using 1DSVM for Stuttgart's wind speed and direction, respectively, with respect to the corresponding month using 1DRF.At the same time for Netherlands's, total accuracy using the 1DLSTM is enhanced by up to 7.1% and 5.4% (for March month) and using 1DSVM by 6.9% and 5.5% (for October month) for wind speed and direction, respectively, with reference to the corresponding month using 1DRF.Accuracies achieved in this work can be further improved with ensemble of the proposed algorithms with the recent advanced machine learn-ing algorithms.The future focus for the authors will be to improve the proposed algorithms and utilising the outputs on interactive visual analysis dashboard applications.Meanwhile, predicted speed and direction has the potential for selecting a feasible location for wind turbine installation, as the turbine power output is highly dependent on the wind speed and direction and moreover provide a foreknowledge for the better city planning.

Figure 1 :
Figure 1: Learning curves for testing samples.

Figure 2 :
Figure 2: Total accuracy for different months for Stuttgart (in speed case).

Figure 3 :
Figure 3: Total accuracy for different months for Stuttgart (in direction case).

Figure 4 :
Figure 4: Total accuracy for different months for Netherlands (in speed case).

Figure 5 :
Figure 5: Total accuracy for different months for Netherlands (in direction case).

Figure 6 and
Figure 6 and Figure 7 shows precision and recall values of dominant speed and direction prediction for June month of Stuttgart.Similar results were obtained for other months as well.Maximum precision and recall values are 92.6% and 92.7% respectively using 1DRF, 92.4% and 93.1% respectively using 1DSVM and 93.7% and 93.4% respectively using 1DLSTM.Further, value of V b was varied to carry out its sensitivity analysis.In first case, V b were varied and V f remains same as V b .In second case, V f was kept constant at 50 while V b varies.These were performed for June month of Stuttgart and total accuracy for dominant speed

Figure 6 :
Figure 6: Precision values of different classes for June month of Stuttgart.

Figure 7 :
Figure 7: Recall values of different classes for June month of Stuttgart.

Figure 8 :
Figure 8: Total accuracy variation for different V b , with V f = V b .

Figure 9 :
Figure 9: Total accuracy variation for different V b , with V f = 50.

Table 1 :
Various classes designed ranges.Class Lower Range Upper Range 1