A B AYESIAN A PPROACH B ASED ON A CQUISITION F UNCTION FOR O PTIMAL S ELECTION OF D EEP L EARNING H YPERPARAMETERS : A C ASE S TUDY WITH E NERGY M ANAGEMENT D ATA

,


A b s t r a c t
With the recent rollout of smart meters, huge amount of data can be generated on hourly and daily basis.Researchers and industry persons can leverage from this big data to make intelligent decisions via deep learning (DL) algorithms.However, the performance of DL algorithms are heavily dependent on the proper selection of parameters.If the hyperparameters are poorly selected, they usually lead to suboptimal results.Traditional approaches include a manual setting of parameters by trial and error methods which is time consuming and difficult process.In this paper, a Bayesian approach based on acquisition is presented to automatic selection of optimal parameters based on provided data.The acquisition function was established to search for the best parameter from the input space and evaluate the next points based on past observations.The tuning process identifies the best model parameters by iterating the objective function and minimizing the loss for optimizable variables such as learning rate and Hidden layersize.To validate the presented approach, we conducted a case study on real-life energy management datasets while constructing a deep learning model on MATLAB platform.A performance comparison was drawn with random parameters and optimal parameters selected by presented approach.The comparison results illustrate that the presented approach is effective as it brings a notable improvement in the performance of learning algorithm.

R e s e a r c h O b j e c t i v e s
One of the key challenges in the implementation of deep learning (DL) algorithms is the correct settings of its parameters to achieve optimal results.The need for tuning DL parameters is ubiquitous in engineering industry as they heavily influence the performance of learning algorithm.In practice, the process of finding good parameters is computationally expensive and complex task.The existing approaches (1-3) include a manual search of parameters which is time-consuming and complex process.The methods like random search (4), grid search ( 5), evolutionary search (6) and guided search ( 7) are used heavily in the past.However, these methods are relatively inefficient for the automatic selection of parameters as they do not choose the next hyperparameters based on previous results.Additionally, these methods need a significant amount of time for calculation of parameters which is not desirable in practical applications.In this study, a practical approach based on Bayesian optimization and acquisition function was established for the automatic selection of DL parameters.The study aimed at identifying the best feasible parameters while conducting the experiments with real-life energy management datasets.

M e t h o d o l o g y
This study begins with the construction of objective function in MATLAB environment for hyper tuning of parameters.Two key parameters i.e hidden layers size and the learning rate are considered during experiments as they mainly influence the performance of the algorithm.The tuning process follows the concept of Bayesian optimization based on acquisition function.
The prior distribution of parameters was formed during the optimization process and acquisition function utilizes this distribution to systematically decide the new set of hyperparameter configurations.To determine the best parameters, K-fold cross-validation (8) was constructed to determine the loss at each iteration.The goal of hypertuning proocess is to obtain the set of hyperparameter values that can give the lowest Root Mean Square Error (RMSE) for the deep learning model.The parameters with the lowest cross-validation loss are considered as optimal parameters and they are selected to train the final model.To validate the presented approach, a typical deep learning model was constructed in a MATLAB environment and performance is compared with random and tuned parameters.We theoretically analyze our method by using three real-world datasets.

R e s u l t s
The results obtained from the presented approach are tabulated in Table 1.A deep learning model is trained with random and tuned parameters and performance is measured in terms of RMSE.For a fair comparison, we used 4 K-fold cross-validation and iterated the results for 4 times.The average of these results concluded that the proposed tuning method achieved better results compared to manual tuning.The RMSE is lower on all three datasets.Additionally, we noticed a reasonable average percentage improvement for Dataset 1 (Home A), Dataset 2 (Home B) and Dataset 3 (Home C) with RMSE of 32.34 %, 47.55 %, 79.69 % respectively.x 100

Findings
The results of three real-world datasets validate that the presented scheme is statistically significant compared to the manual tuning of parameters.The investigation has lead to several benefits.For instance, it can: • reduce the human intervention required for the implementation of DL algorithms.In power engineering applications, this is particularly crucial.
• enhance the accuracy of DL algorithms by iteratively finding the globally optimal parameters.
• offers the opportunity to automate the deep learning parameters in real-time systems as it achieves a good set of parameters in a small number of experiments.

Table 1 :
Comparison of performance with random parameters and presented approach