German country-wide renewable power generation from solar plus wind mined with an optimized data matching algorithm utilizing diverse variables

Abstract

Country-wide, hourly-averaged solar plus wind power generation (MW) data (8784 data records) published for Germany in 2016 is compiled to include ten influential variables related weather, ground-surface environmental and a specifically calculated day-head electricity price index. The transparent open box (TOB) learning network, a recently developed optimized nearest neighbour, data matching, prediction algorithm, accurately predicts MW and facilitates data mining for this historical dataset. The TOB analysis results in MW prediction outliers for about 1.5% of the data records. These outliers are revealed via TOB analysis to be related to uncommon conditions occurring on a few specific days typical over hourly sequences involving rapid change in weather-related conditions. Such outliers are readily identified and explained individually by the TOB algorithm’s data mining capabilities. A slightly filtered dataset (excluding 129 identified outliers) improves TOB’s prediction accuracy. The TOB algorithm facilitates accurate predictions and detailed evaluation over a range of historical temporal scales on a country-wide basis that could also be applied to regional spatial predictions. These attributes of the TOB method are conducive with it eventually being incorporated into forward-looking renewable forecasting frameworks.

Appendices

Appendix A: dataset analysed

The compiled pre-processed hourly-average dataset analysed in this study is included as a supplementary file. The underlying data is published online and freely available in two distinct OPSD databases [52–54], but is not assembled there in the convenient hourly-averaged form for each variable as compiled and used in this study.

Appendix B: transparent open box algorithm calculation steps

The fourteen steps of the transparent open-box (TOB) algorithm [73, 74] are outlined here.

2.1 TOB Stage 1 (initial prediction applying unweighted data matching)

Step 1 Data is arranged in a 2-D array (M data records; N independent variables plus one dependent variable, i.e., the variable to be predicted, i.e., megawatts of power generated in this study).

Step 2 Data records are sorted into ascending or descending order of the prediction variable’s values.

Step 3 Each variable is expressed in basic statistical terms to characterize the variable distributions involved in the dataset to be evaluated (e.g., Table 1).

Step 4 Each variable’s maximum and minimum values are used to normalize the dataset over a scale of − 1 to + 1 by applying Eq. (10).

$$ X_{i} * = \, 2 \times \left[ {\left( {X_{i} - \, Xmin} \right)/\left( {Xmax - Xmin} \right)} \right] - 1 $$

(10)

where X_i= the ith data record for X of N + 1 variables, Xmin = minimum of variable X for the entire dataset, Xmax = maximum of variable X for all data records, X_i* = normalized value for the ith record for X of N + 1 variables.

Step 5 Verify the values of the dataset confirming that all variables are indeed correctly expressed in normalized terms i.e., − 1 ≤ X_i* ≤ +1.

Step 6 Allocate all the data records in the dataset to either a training subset, a tuning subset or a testing subset. This allocation is best not performed in a random manner to ensure that the full variable range is sampled by the tuning and testing subsets. Arranging the data records in ascending or descending order of dependent variable values (Step 2) enables the dataset to be sampled repeatedly at different offset intervals across the full value range when populating the tuning and testing subsets. Sensitivity analysis is required to establish the optimum allocations to each subset that provide the most accurate predictions. Typically, the training subset is likely to constitute more than seventy percent of the dataset records. The two-stage process of the TOB facilitates such sensitivity analysis. A series of TOB cases are run with different sizes of tuning subsets. Comparing the prediction performances of TOB stages 1 and 2 identifies the minimum number of data records needed in the tuning subset for the Stage 2 TOB predictions to consistently outperform the stage 1 predictions.

The requirement for separate tuning and training subsets is to enable the optimizer to tune the weights applied to the squared errors of each independent variables in the records of the training. This tuning approach can provide accurate predictions for a representative, but relatively small number of data records in the tuning subsets. For large datasets (e.g., many thousands of data records and multiple variables) large tuning subsets (e.g., substantially greater than about 150) tend to increase the computational effort without providing any benefits to the optimization process of TOB stage 2. By focusing on relatively small tuning subsets computational effort is reduced.

Step 7 Compute the variable squared error (VSE) for each of J tuning-subset records versus the K training-subset records using Eq. (11):

$$ VSE\left( X \right)_{jk} = \left[ {X_{k} \left( {tr} \right) - X_{j} \left( {tu} \right)} \right]^{2} $$

(11)

where $X_{k} \left( {tr} \right)$ = variable X value for the kth training-subset record, $X_{j} \left( {tu} \right)$ = variable X for the jth tuning-subset record, $VSE\left( X \right)_{jk}$ = variable-squared error (VSE) for variable X for the jth tuning-subset record versus the kth training-subset record. ∑ $VSE_{jk}$ = weighted sum of the computed VSE values applying Eq. (12):

$$ \sum VSE_{jk} = \mathop \sum \limits_{n = 1}^{n = N + 1} VSE\left( {Xn} \right)_{jk} \times \left( {Wn} \right) $$

(12)

where $VSE\left( {Xn} \right)_{jk}$ = the variable-squared error (VSE) for variable Xn for the jth tuning-subset record versus the kth training-subset record. $\sum VSE_{jk}$ = sum of variable-squared errors (VSE) for the N + 1 variables (including the dependent variable) for the jth tuning-subset record versus the kth data training-subset record.

Wn = weights (0 < Wn <= 1) applied to the calculated VSE for all variables involved in the prediction (i.e., N + 1). Each weight is set to a constant value (e.g., 0.5 is used in this study) in TOB stage 1. This avoids any bias being introduced into the ranked initial matches derived for the tuning versus training subsets.

Step 8 Rank the matching data records in the training subset versus each tuning-subset record. The training-subset record that possesses the smallest calculated ∑VSE value is identified as the best matching record for a specific tuning-subset data record. The top-Q-matching training-subset records, established for each tuning-subset record, are then selected for the initial TOB-stage-one prediction. Q = 10 has empirically been found to be sufficient to provide accurate TOB-stage-one predictions for a wide range of datasets. Increasing Q much above ten adds to the computational effort without providing benefits of sufficient improvements in prediction accuracy.

Step 9 The top ten, best matching records in the training subset for the jth tuning-subset record each contribute fractionally to the TOB-stage-one prediction for that record. The exact contribution fraction applied to each of those top-ten matching data records is established with Eqs. 13–15. The computed contribution fraction is calculated using the ∑VSE values for each training-subset record versus the jth tuning-subset record.

$$ f_{jq} = \sum VSE_{jq} /[\mathop \sum \limits_{r = 1}^{r = Q} \sum VSE_{jr} ] $$

(13)

where q = the qth of Q top-ranking training-subset records from the training subset for the jth tuning subset record. r = the rth of Q top-ranking training-subset records from the training subset for the jth tuning subset record. f_q= the contribution fraction calculated for the qth of Q top-ranking records for the jth tuning subset record.

Equation (14) adjusts the f_q values to sum to 1.

$$ \mathop \sum \limits_{q = 1}^{q = Q} f_{q} = 1 $$

(14)

The best-matching training-subset record (i.e., the one with the lowest $\sum VSE_{jk}$ value) should make the greatest contribution to the prediction of the dependent variable associated with the j^th tuning-subset record. This is achieved by applying $\left( {1 - f_{q} } \right)$ multipliers in Eq. (15).

$$ \left( {X_{N + 1} } \right)_{j}^{predicted} = \mathop \sum \limits_{q = 1}^{q = Q} \left[ {\left( {X_{N + 1} } \right)_{q} \times \left( {1 - f_{q} } \right) } \right] $$

(15)

where: $\left( {X_{N + 1} } \right)_{q}$ = dependent variable for the qth training-subset record (i.e., one of Q best-matching records). $\left( {X_{N + 1} } \right)_{j}^{predicted}$ = TOB-stage-one predicted-dependent-variable value for the jth tuning-subset record.

The output from Step 9 represents the TOB-stage-one prediction. It is provisional because it applies equal weights (Wn) to the variable squared errors (VSE). This prediction is further refined in TOB stage 2 optimization.

Step 10 Various statistical metrics are used to assess the accuracy of the TOB-stage-one predictions (see Sect. 3.3).

2.2 TOB Stage 2 (optimizing contributions from the best matching records)

Step 11 Optimized predictions are achieved by minimizing the root mean squared error (RMSE) metric (Eq. 3, Sect. 3.3). RMSE is computed using the squared prediction errors calculated for all J tuning-subset records. Three optimizers are run in this study to verify the results and generate a range of potentially optimum solutions (see Sect. 3.2).

Two TOB stage 2 minimization control metrics are Q and W_n are employed:

• 1. The N input-variable weights (W_n) are allowed to vary across the fully constrained range (0 ≤ Wn ≤ 1) leaving the optimizer free to select the best values for them to minimize RMSE. It is not unusual for quite small non-zero values to be assigned to certain W_n by the optimizer. However, these low weights often have significant impacts that improve the prediction accuracy of TOB stage 2.

• 2. The optimizer is allowed to vary Q (2 ≤ Q ≤ 10) for the TOB stage 2 calculations conducted with Eqs. 13–15.

Step 12 Statistical accuracy metrics for the TOB-stage-two predictions are computed and compared with those generated for the TOB-stage-1 predictions. Performing sensitivity analysis by optimizing with different fixed values of Q (i.e., Q = 2 to 10) also generates a set of sub-optimal solutions that also help assess potential underfitting or overfitting issues with the data set being evaluated.

Step 13 Compute TOB-stage-one and TOB-stage-two predictions for the independent testing subset records applying the optimum values established for W_n and Q in step 11 established by tuning the tuning subset Assess these predictions with the statistical accuracy metrics and compare them with those obtained for the tuning subset (e.g. Tables 3 and 5). They should be in comparable ranges. If they are not it is indicative that the tuning process may be either underfitting or overfitting the dataset. Further verification of the TOB Stage 2 optimum solutions prediction performance can be obtained by applying those solutions to the all records in the dataset and assessing the prediction accuracy achieved (e.g., Table 6).

As part of this step it is often appropriate to audit the intermediate calculation steps to reveal which variable errors (VSE) are weighted to contribute the most to the TOB Stage 2 predictions. Reviewing the intermediate calculations can also facilitate comprehensive outlier analysis (e.g., see Sect. 5.1). This provides insight as to why some data records lead to less-accurate predictions than the main trend of predictions). Also segmental analysis (see Sect. 4.3) identifies regions of the dependent-variable range for which the TOB method provides greater or lesser prediction accuracy.

Step 14 Compare the prediction accuracy provided by the TOB algorithm with other prediction methods (e.g., regression-based, machine learning algorithms and empirical methods). Such analysis for the renewable power generation dataset studied here will be presented in future studies.