Dual Heuristic Neural Programming Controller for Synchronous Generator

This chapter describes the development of voltage control system of a synchronous generator based on neural networks. Recurrent (dynamic) neural networks are used, as a type that has great capabilities in approximation of dynamic systems. Two algorithms are used for training – Dual Heuristic Programming (DHP) and Globalized Dual Heuristic Programming (GDHP). The algorithms have been developed for the optimal control of nonlinear systems using dynamic programming principles.


Introduction
This chapter describes the development of voltage control system of a synchronous generator based on neural networks.Recurrent (dynamic) neural networks (RNN) are used, as a type that has great capabilities in approximation of dynamic systems [1].Two algorithms are used for training -Dual Heuristic Programming (DHP) and Globalized Dual Heuristic Programming (GDHP).The algorithms have been developed for the optimal control of nonlinear systems using dynamic programming principles.
Neural voltage controller is developed in MATLAB and Simulink environment.For training purposes a mathematical model of synchronous generator is designed and applied in Simulink.DHP and GDHP algorithms are designed in Simulink, with matrix calculations in Sfunctions.Algorithms are used for offline training of neural networks (NN).In the second part, the same functions are redesigned as real time controllers, based on the Real Time Windows Target Toolbox.DHP or GDHP algorithms provide a significant improvement over conventional PI controller with, in some cases, power system stabilizer (PSS).Conventional linear controller can only provide optimal control in single operating point.Algorithms described in the chapter provide significantly better control over the whole operating range by minimization of the user defined cost function during the training of the neural network.
Digital voltage controller is implemented on real system in two basic steps.First step is neural network design and training in Matlab environment on desktop computer.Second step consists of transfer of controller with computed coefficients to the digital control system hardware.
For the controller to have optimal performance in all real operating conditions, neural networks must be trained for the whole working range of the plant.Procedure described in the chapter can be applied on standard voltage control systems by replacing the PI controller with the offline-trained neural network.Most modern digital control systems have sufficient hardware and software support for neural network implementation.

Recurrent neural networks
Neural networks are computational models capable of machine learning and pattern recognition.They can also be used as universal approximators of continuous real functions.Typical dynamic neural network is shown in Figure 1.Typical activation function maps input values u(t) ∈ − ∞, ∞ to output y(t) ∈ 0, 1 or y(t) ∈ − 1, 1 .Activation function must be derivable over the whole range.Among typically used nonlinear activation functions are logsig, tansig and Gaussian.Dynamic behavior is added by introducing a memory element (step delay) that stores neuron output and reintroduces it as input to all neurons in the same layer in next time step.This type of network is named recurrent or dynamic neural network.An example of tansig (Hyperbolic Tangent Sigmoid Transfer Function) activation function, ϕ ( u ) = 2 1 + e −2u − 1, and its  Use of function with simple derivative significantly improves calculation times of numerical procedures.Dynamic (recurrent) neural networks provide good approximation of time-dependent functions.However, training of recurrent neural networks is more complex than training static networks.

Recurrent neural networks training
There are many methods for training neural networks, and most of them rely on some form of backpropagation -calculation of partial derivative of system output over network weight coefficients ( ∂ y ∂ w ).( ) where p 1 -hidden layer input, p 1 = u(t) ; x(t − 1) ; x t f x t p t w t w t w t (5) where N in -number of neurons in input layer.
Value of ∂ x l (t) ∂ w i, j is calculated using four nested loops (Appendix A).
Matrix of output derivatives over weight coefficients is of the form

Gradient descent
Gradient descent is the most commonly used method in neural networks training.Difference between real and desired network output can be expressed as Differentiation of ( 5) provides update values of weight coefficients: y t w t w t w t (7) Parameter β defines learning rate and must be in range 0 < β < 1.
( ) Training based on Kalman filter is fast, has good rate of convergence and is often the only practical solution for recurrent networks training.However, the procedure is demanding in terms of computational power and memory requirements.

System identification
Nonlinear dynamic systems are identified as input-output system model, or as a state space system.For the selected process, identification is performed by minimization of the cost function T E t y t y t y t y t (11) where y(t) − y ^(t) is the difference of the measured system outputs and output values of the model.
Selected cost function represents squared error of the identified model.
Dynamic system identification can be performed according to [9], using either series-parallel or parallel structure.

Non linear dynamic system
Neural Networks

Non linear dynamic system
Neural Networks In series-parallel structure Figure 3, model neural network input vector is expanded with system outputs from previous steps.In parallel structure Figure 3, network output from previous states is added to input vector.Series-parallel identification provides estimated system output for next time step only, while parallel identification can predict multiple future iterations.However, parallel structure is harder to train and is susceptible to state drifting.
To verify the validity of the RTRL algorithm, neural networks were trained on several discrete mathematical functions.Supervised training has been used, with control value expressed as difference between desired and actual network output.
A parallel identification structure was used on dynamic SISO system y(t + 1) = y(t) 1 + y(t) 2 + u(t) 3 .Training has been performed on Recurrent Neural Network with four neurons in hidden layer, and one neuron in linear output layer.Identification results are shown in Figure 4.In second test, a system with one input, two state variables and one output has been identified: System was identified using series-parallel structure with RNN with 5 neurons in hidden layer, and one linear neuron in output layer.Identification results are shown in Figure 5. Input signal used for training and verification is the same as in previous example: Both examples were designed in Matlab and Simulink.Neural network was trained using clanguage S-function (Appendix B).Although the training took 10000 steps, all important system dynamics were visible on trained network after 2000 steps.

Determination of partial derivatives of functions using neural networks
One of the most important features of neural network modeling is the possibility to determine system output derivatives over inputs ( ∂ y i (t) ∂ u j (t) ).After finding output derivatives, various optimization algorithms can be used.
Partial derivative is calculated from (2) (12) Dual Heuristic Neural Programming Controller for Synchronous Generator http://dx.doi.org/10.5772/58377 where For nonlinear function was first calculated analytically, and then obtained from model network using (12).Results are compared in Figure 6a and 6b.Results from Figure 6b show good match between calculated function derivatives and values obtained from neural network of identified model.
In S-function for training RNN (Appendix B) a procedure is shown for online calculation of partial derivatives of ( ∂ y i (t) ∂ u j (t) ), using (12).

Adaptive critic design
Adaptive critic design (ACD) is a set of optimal control procedures of nonlinear systems.It is based on dynamic programming and neural networks.Optimal control is achieved by training neural networks and using them in the structure according to the selected ACD algorithm.ACD algorithms are implemented using a heuristic approach -system state space variables, control values and cost function are selected based on process characteristics.This text focuses on the Heuristic Dynamic Programming (HDP), Dual Heuristic Programming (DHP) and General Dual Heuristic Programming (GDHP) [1,2].

Heuristic dynamic programming
Heuristic dynamic programming (HDP) is the simplest (by structure) ACD algorithm.The basis of all HDP algorithms is minimization of Bellman dynamic programming function J (t): where γ represents discount factor with value in range 0 < γ < 1, ensuring the convergence of Bellman sum J (t) and cancellation of old values, ) is user defined utility function based on system state space variables.
Control signal that minimizes Bellman sum is generated using one of the Adaptive Critic algorithms.In practical control systems, HDP optimization is realized with three neural networks with a structure as shown in Figure 7.
Model neural network in 7a is used for model identification.Network inputs are control signal u(t) and output state in the previous time step y(t − 1).Network output represents estimated model output in the current time step y ^(t).Only systems outputs used in utility function need to be estimated.Model NN is structured according to the selected process model, where the control value is input to the NN, and NN output represents predicted system output.The same NN output values are used to calculate the cost function.
Model neural network is trained to minimize the quadratic criterion: where: For adaptive control systems, model NN is trained continuously, allowing for adaptation to change of system parameters, and optimal control.Model neural network is trained simultaneously with other networks over time.
Critic neural network in 7b is used for identification of Bellman sum J (t).Training is used to minimize where: [ ] Critic neural network training is performed with a copy of the same network which is fed with predicted future system outputs y ^(t + 1), thus approximating J (t + 1).
Action neural network training is based on model and critic network outputs, by minimizing criterion: where:  7c.

Dual heuristic programming
Dual Heuristic Programming optimal control algorithm is developed with a goal of increasing procedure efficiency by increasing training speed.
Model network training is the same as in HDP procedure.The improvement is in the direct calculation of the Bellman sum partial derivative over state space variables ∂ J (t + 1) ∂ y(t) as output of critic network.
Structure of DHP control system is shown in Figure 8.
Training of critic neural network minimizes scalar product: ( ) Third part of equation ( 22) is determined from deriving Action neural network is trained to minimize Bellman sum, min J (t) , from the criterion Thus, equation ( 22) can be transformed to Training of Action Network minimizes scalar product: Simplified training procedure of action NN is shown in Figure 8a.
Cross training of critic and action networks results in slower approximation.In order to reduce training time, it is recommended to begin the procedure with action network pre-trained as linear controller.

Global dual heuristic programming
Globalized Dual heuristic optimal control algorithm uses both algorithms described.
Critic Action network is trained using the same procedure as described for HDP and DHP.

Neural networks based control of excitation voltage
Main elements of electric power system are generators, transmission and distribution network and consumers.Transmission and distribution network form a power grid, which is usually very large in geographic terms.Grid control is implemented on both local and global level, mostly on the power generation side of the grid.Specific to power grid control is large number of closed control loops governing the power and voltage of synchronous generators.Transmission network is governed by setting voltage on transformers in order to minimize transmission losses.Governing of generator power and voltage achieves balance of power as well as reactive power flow on the transmission system.Voltage control also has a large impact on the dynamic system behavior.
Generator power output is governed on the generator unit.For water and steam turbine generators, it defines local power transmission to the grid, while also influencing system frequency.Considering the inertness of the generator systems, power output is not difficult to govern.Voltage control manages the balance between generator and grid voltage, and determines reactive power flow.A change in generator voltage setpoint has impact on system state variables and parameters.Disregarding system transients, a change of generator voltage causes a change on generator busbars, generator current and reactive power.
If we also consider system dynamics, generator voltage change also changes system parameters influencing system damping.Synchronous generator connected to the power grid is dominantly nonlinear in characteristics.Generator output power is proportional to generator and grid voltage, while the relation to angle between generator and grid voltage is a sinus function.Maximum theoretical output power is realized with a load angle being δ = π 2 .
Exceeding of maximum value of load angle causes large disturbances and disconnection from power grid.
Operating point of generator connected to the grid is determined by steady state excitation current.Excitation voltage and current values dictate generator voltage, reactive power and load angle.
Frequent small disturbances expected and part of normal operating conditions, resulting in oscillations of generator electromechanical variables.Especially pronounced is the swinging of generator power, reaching several percents of nominal value, with a frequency around 1Hz.It is caused by low system damping and outer disturbances on the grid.Also present on the grid are oscillations caused by the system characteristic frequencies, usually around 0.1Hz.
Power grid can be represented as a single machine connected to an infinite bus system.For each operating point a characteristic frequency and damping can be determined.
In case of small disturbances with a synchronous generator working around predefined operating point, system can be considered linear, but only if it returns to the same operating MATLAB Applications for the Practical Engineer point after each disturbance.For those conditions, power system stabilizer (PSS) is added to AVR.Based on change of output power and load angle, it modifies voltage regulator setpoint and increases system damping.PSS is configured using linear control theory, and is presently a standard part of automatic voltage regulator systems [10].
Large disturbances occur after short circuits on the power grid, transmission and generator outages, and connection and disconnection of large loads.
Electromechanical oscillations occurring as a consequence of disturbances cause damped oscillations of all electrical and mechanical values (generator voltages and currents, active and reactive power, speed and load angle).Oscillations are also propagated by AVR to excitation system values.
Large system disturbances can cause significant changes to local parameters relevant for generator stability.Due to the linear nature of the voltage regulator system, a change in grid voltage or configuration leads to generator working in non optimal state.Even if the system returns to the same operating conditions, passing through nonlinear states during disturbance causes non optimal control, diminishing the stabilizing effects of the voltage regulator on the system.This can in turn cause another disturbance or, in case of larger units, disruption of the whole power grid.Limitations of the use of linear control systems call for an improved system which can provide optimal control of generator excitation system [13]- [15].
There is a lot of research effort directed toward design of excitation control system based on neural networks.However, very few of the demonstrated solutions have been implemented on real plants.
The goal of the chapter is to develop a practical-to-use system.This is mainly achieved by separating the design on preparation and implementation phase.Almost all algorithm steps are executed in off-line mode, up to the final step of digital voltage regulator implementation.
To implement the voltage regulator described in the chapter following equipment is needed: • data acquisition hardware and software

• personal computer with Matlab and Simulink environment
• automatic voltage regulator with neural network support (no training support is needed) Described controller is not adaptive.Adaptive critic design (ACD) algorithm is used for optimal control.
Classic voltage regulators (AVR) equipped with PSS can be very effective in voltage control and power system stabilization if it is used in well configured transmission grid.In such conditions the generator stays in the linear operating range and the controller maintains optimal configuration on change of active and reactive power.Voltage regulator accuracy, balance of reactive power, stabilizing effect during large disturbances and active stabilization in normal operating conditions are achieved.
If the generator is connected to relatively weak transmission grid, with possible configuration changes during operation, use of neural network based control of excitation voltage leads to better results.

HDP algorithm implementation
In order to achieve optimal control of synchronous generator connected to the power grid, it is necessary to select a minimum of two system variables -generator voltage and either output power or angular velocity.Selected variables must reflect system oscillations during disturbances.It is also necessary to choose cost function that is minimized during training.System optimization is achieved by minimization of squared generator voltage governing error ( ) ( ) It is important to notice that the two demands are in conflict.Increase in gain in main control loop reduces static control error, but also reduces system damping and increases power output deviation ΔP.Use of cost function (27) ensures training process that increases system damping without sacrificing controller responsiveness.
System identification is carried by training model neural network.Figure 8 shows training process of model neural network.Identification of nonlinear system is made by series-parallel structure.
For ACD algorithm optimal control two output state space variables are chosen -generator voltage and output power, y = ΔU G ; ΔP .Vector y is used to form the cost function.Input Model NN has 9 inputs and 2 outputs.
Figure 9 shows simulation block diagram of identification procedure, implemented in Matlab and Simulink environment.

Figure 9. Model identification block diagram
A third order nonlinear mathematical model was used as a synchronous generator [20].Synchronous generator is modeled as a single machine connected to an infinite bus.Although the model ignores initial magnetic flows, it proves to be a good approximation of an actual generator.A predictor-corrector method is used to solve differential equations.
Selected mathematical model, used with well conditioned state space variables and described methods, achieves good numerical stability during simulation, allowing for relatively large sample time (T D =0.02s).Described model of synchronous generator is implemented in Clanguage S-function and used for further simulations.
For model identification, recurrent neural network is used, with five neurons in hidden layer and two neurons in output layer.Hidden layer is implemented with full recursion and tansig activation function.Output layer uses linear activation function.
Initial learning rate value η In the same experiment efficiency of the algorithm was verified.Training was stopped after 100000 steps.For the rest of the simulation, network output is calculated without the change of weight coefficient, thus eliminating the recency effect.[ dY2du] [ dY1du] [dUdu] [Pg] [Ug] [ dY2du] [ Pg] [ dY1du] [dUdu] [Ug] [ dY2du] [ dY1du] [Pg] Training results are shown in Figure 12. Figure 12a compares generator voltage response using conventional automatic voltage regulator (AVR) to neural network based controller trained using HDP algorithm.It can be seen that voltage rate of change was not degraded as a result of system damping.It is important to emphasize that generator voltage rate of change is key to system stability during large disturbances.
Figure 12b compares active power output oscillation of conventional and HDP controller.
Figure 12c shows load angle, which is indicative of system stability (δ< π 2 ).System damping is significantly increased by use of neural network controller.Figure 13d shows conventional automatic voltage regulator (AVR) operating on the edge of stability, while the DHP controller is completely stable.Comparison of load angle values demonstrates significantly better behavior of DHP controller.On large load angles DHP controller maintains stability while conventional AVR provides low system damping, leading to instability and possible outages.
Neural network HDP voltage controllers were developed using DHP algorithm.Experimental results showed that DHP algorithm is more efficient than ADC algorithm.Implementation of GDHP does not bring significant improvements over DHP.
Application of developed S-functions enables simple use of described algorithms, as well as experimenting with different function parameters.

Real time windows target implementation
DHP optimal control algorithm has also been implemented using Matlab Real Time Windows Target Toolbox (RTWT), based on [18] and [19].Simulation hardware consists of desktop PC equipped with National Instruments data acquisition card NI DAQ 6024.
Use of RTWT platform enables direct implementation of developed controllers on the real system.
In Figure 14 ACD algorithm is implemented in two steps.Classic voltage controller is used in the first step (Figure 14).In the second step it is replaced with NN controller.
During testing, generator active power and voltage were changed for a wide range of values.
A pulse signal was superimposed to the generator voltage set-point value, resulting in large disturbances on the generator.To obtain measurement records RTWT Signal Visualization was used, with measured data saved in Workspace Simulink block yP.Recorded data was used to train neural network in offline mode.
A RNN was used with one hidden layer containing 5 tansig neurons.Model NN had 9 inputs and 2 outputs.Results of NN training are displayed in Figure 15.Obtained model NN was used to train critic and action RNN using procedure described in 3. The trained action NN was transferred to Simulink block (Figure 14) and prepared for use in real time.Using RTWT module system was run in real-time mode, with trained action RNN as a voltage controller.Figure 16a represents comparison between generator voltage responses and shows satisfactory rate of generator voltage change and high control accuracy.In Figure 16b, generator active power outputs are compared.Considerable damping was achieved using DHP controller.Improved system stability during large disturbances is visible in Figure 16c, where the maximum load angle deviation is significantly lower for DHP controller.
RTWT controller was implemented in sample-time mode with sample time set to T s = 2.5 ⋅ 10 −4 s.Control loop sample time is T D = 0.02s.
Matlab Simulink RTWT Toolbox simplifies the transfer of developed procedures to the hardware platform.

Figure 1 . 1 )
Figure 1.Recurrent neural network structureNeural network is formed from interconnected layers of neurons.Output of each neuron is calculated from:

Figure 4 . 1 Figure
Figure 4. Identification of dynamic function 1 Figure 4a compares initial response of model output y(t) and network output y ^(t).Figure 4b represents the same values with trained network.There is a perfect match between function output values and output of the trained NN. Figure 4c displays training rate as error square ( y(t) − y ^(t) ) 2 .Training was completed in 20000 steps.Input signal used for training and verification is u(t) = sin( 2π 50 ) + sin( 2π 12 ), with sample time (T D = 1s).

Figure 6 .
Figure 6.Identification of nonlinear function and calculation of partial derivatives using neural networks

Figure 8 .
Figure 8. DHP structure, (a) Critic NN training, (b) action NN training and minimization of output power change ΔP obtained from: (21) is set to η = 10 5 .It was reduced during training to η = 10 2 .Training was completed in ~10000 steps.Results are shown in Figure 10.Signals U G NN(0) and U G NN(1) represent generator voltage at the start and at the end of RNN training.Figure 10a compares responses of control error of generator model and neural network.Voltage setpoint is generated as a pulse signal with a magnitude of ± 10%.In figure 10b, response to the change of generator power is shown.Figure 10c shows squared sum of RNN error during training.

Figure 10 .
Figure 10.Training results of model neural network Once the model neural network is trained, it is possible to create simulation for training critic and action neural networks.Simulink model of the simulation is shown in Figure 11.

Figure 11 .
Figure 11.Simulink model for training action and critic networks

Figure 12 .
Figure 12.Response comparison of conventional AVR and HDP controller.a) generator voltage; b) generator active power; c) load angle

Figure 13 .
Figure 13.Response comparison of conventional AVR and HDP controller a), b) generator voltage; c), d) generator active power; e), f) load angle Figure 13c and 13e show comparison of output power and load angle change.It is obvious that the HDP neural network controller shows significant improvement over classic regulator.

Figure 13b shows comparison
Figure 13b shows comparison of voltage transient in the condition generator voltage reduced to 90% of nominal value.Rate of change of generator voltage is maintained even during such extreme operating conditions.
, a system consisting of synchronous generator and RTWT controller is shown.Signal acquisition is achieved using DAQ 6062E Input, with AD conversion of the input values.Line voltage and phase current signals are processed in "Clark" S-function, using Clarke transformation.Transformed values of generator voltage U G and generator power P are used as control feedbacks.Digital to analog conversion is performed by output module DAQ 6062 Output.All simulation blocks are part of a standard Simulink library except Clark transformation and RNN S-functions.

Figure 15 .Figure 14 .
Figure 15.Results of Model NN training a) generator voltage; b) generator active power

Figure 16 .
Figure 16.Classic controller and HDP NN controller -system responses: a) generator voltage; b) generator active power; c) load angle Simulation results are shown in Figure 16, comparing classic and DHP voltage controllers.Figure16arepresents comparison between generator voltage responses and shows satisfactory rate of generator voltage change and high control accuracy.In Figure16b, generator active power outputs are compared.Considerable damping was achieved using DHP controller.Improved system stability during large disturbances is visible in Figure16c, where the maximum load angle deviation is significantly lower for DHP controller.
Partial derivatives are determined using chain rule, by finding derivatives of weight coefficients in output layer, using intermediate results to calculate next layer, with respect to network structure.Calculation is repeated until all layers are processed.Described procedure is valid for static neural networks, but is of no use in recurrent networks, as each neuron has inputs from previous time step from the same layer.Two procedures are often used in calculation of partial derivatives: Backpropagation Through Time (BPTT) and Real Time Recurrent Learning (RTRL).
In BPTT recurrent neuron inputs are replaced with the same neural network delayed by one time step.Process is iterated N times.Thus, recurrent network is approximated by N feedforward networks with delayed outputs.Network weight coefficients, recurrent inputs and outputs must be preserved for past N time steps.Obtained derivative ( ∂ y ( t ) ∂ w ( t ) ) is used to determine new weight coefficients.One advantage of BPTT is that all methods used for training feedforward networks can be utilized in training recurrent networks.This chapter will describe use of RTRL procedure in training dynamic networks.Output of recurrent network with one hidden and one output layer has the following form: Dual Heuristic Neural Programming Controller for Synchronous Generator http://dx.doi.org/10.5772/58377 is estimated from critic network.The second part of (21) is determined by deriving Dual Heuristic Neural Programming Controller for Synchronous Generator http://dx.doi.org/10.5772/58377123 HDP algorithm is also verified on voltage control of synchronous generator from Simulink SimPower System block library (Synchronous Machine).Model network is designed with one hidden recursion layer with 5 neurons.Critic and action networks use the same structure, but with four neurons in hidden layer.HDP algorithm shown in Figure 11 is applied.Training of neural networks is performed in two steps-in first step action NN is trained to mimic classic voltage regulator, and in second step DHP algorithm is used to train both action and critic NN.It is important to get initial values of action NN in first step, as DHP needs network partial derivatives in training critic NN. Figure 13 shows training results.Figure 13a shows comparison of classic and NN controller responses.Generator voltage rate of change is almost the same for both controllers.