Real-Time FPGA-RTDS Co-Simulator for Power Systems

This paper proposes a co-simulation platform using field-programmable gate array (FPGA) and real-time digital simulator (RTDS) for the simulation of large power systems. It combines the advantages of high computational power from FPGA and better modelling flexibility from RTDS together. The FPGA therefore acts as an efficient and economical extension to RTDS especially when simulating large ac systems. One of the significant advantages of the proposed co-simulator is that it avoids the potential interface error existing in the conventional approach of interfacing transient stability program with electromagnetic transient programs. Two key aspects of the proposed co-simulator are discussed: 1) the interface design between FPGA and RTDS and 2) the hardware implementation and expandability of the platform. Two case studies are presented to verify the simulation accuracy and capability of the proposed co-simulator. The first case simulates a two-area four-machine power system with one area simulated in FPGA and the other area in RTDS. Comparisons are made with the case where the complete system is simulated in RTDS. The second case simulates a system of 141 buses in FPGA to demonstrate the simulator’s capability in simulating large power systems.


I. INTRODUCTION
Modern electrical power systems are undergoing dramatic changes such as: • Increasing amount of renewable energy integration; • Increasing level of interconnections between regional/national electrical power systems [1]; • Increasing applications of power electronic devices in existing AC networks [2].All these changes result in an increase in size and complexity of power systems.In particular the application of power electronic devices introduces a range of new risks which are different in nature to those in traditional AC power systems.For example they could affect the operation of protection relays [3], cause harmonic problems [4] or control interactions [5], and lead to potential overvoltage/overcurrent problems.Real-Time Simulation (RTS) using Electromagnetic Transient (EMT) type of component models is essential and has been widely used to study these problems [6], [7].
One of the main difficulties for RTS is that significant computational resources are required for the solution of detailed EMT models thereby limiting the size of AC system that can be accurately simulated [6].One of the conventional methods in simulating larger system is to use network equivalents, which neglects internal dynamics of the equivalent part of the system.To increase the size of system that can be simulated while maintaining simulation accuracy, hybrid simulation combining EMT and Transient Stability (TS) programs was proposed [8], [9].This method partitions the whole system into two or more subsystems.The area/areas with power electronic devices is/are simulated accurately using EMT programs.The rest of the network is simulated using TS programs to save computational resources.The accuracy and efficiency of hybrid simulation critically depend on the interface technique [8].Therefore most of the previous works are focusing on the design of interface between TS and EMT subsystems.VOLUME 6, 2018 This work is licensed under a Creative Commons Attribution 3.0 License.For more information, see http://creativecommons.org/licenses/by/3.0/ Conventionally the fundamental frequency equivalent of the TS subsystem is adopted [10] and both positive-sequence and three-sequence equivalent models are used [11]- [13].The drawback of using positive-sequence network is that it will give inaccurate results for unbalanced conditions [11].To mitigate the problem, significant extension of the interface bus was proposed to maintain the assumption of three-phase balance, but simulation efficiency of EMT part is significantly reduced [13].For three-sequence equivalent model, the representation may not be accurate when the transients at interface buses are significant [10].The Frequency-Dependent Network Equivalent (FDNE) was recently proposed in [10] to decrease the interface error.A number of approaches to obtain the FDNE are proposed [14], [15], and different representations of FDNE are adopted [16].The main disadvantages of these methods are the increased computational burden on the EMT simulator and potential inaccuracy under transient conditions.
It is understandable that the design of an interface technique that is accurate and computationally efficient is difficult, as these two simulation methods have fundamentally different mathematical models.
This paper proposes a real-time FPGA-RTDS co-simulator using EMT models only, hence avoiding the complexities in interface design and the potential interface errors exist in hybrid simulation methods.At the same time, the size of the power system that can be simulated is not compromised as a large part of the network can be simulated using FPGA taking advantage of its computational power.The rest part of the network of interest is simulated using RTDS taking advantage of its simulation flexibilities.
It needs to be mentioned that the EMT simulation of large AC network has been implemented using FPGA in [17]- [19].However, the method lacks Graphical User Interface (GUI) for the manipulation of simulation cases and visualization of results.Also the control systems for synchronous generators (excitation system and governor system) are not modelled.Other works have been focusing on the simulation of power electronic devices with small simulation time-step using FPGA [20], [21].In those works, FPGA is used to simulate the detailed model of Modular Multilevel Converter (MMC).One of the difficulties in implementing those methods is to avoid re-synthesize of FPGA (which can take hours) whenever system modification (frequent in simulating power electronic systems) is required.Also the development of a generic model for power electronic device in FPGA can be complex as many of the power electronic devices are still evolving with changing topologies, control strategies and modelling approach [22], [23].
Different from these methods, the proposed method utilizes FPGA to simulate large AC networks rather than power electronic systems, taking advantages of the well-established EMT models of AC components [24], [25].In addition, the flexibility in system reconfiguration, modelling and control that are readily available in RTDS can be utilized.The role of FPGA therefore is to act as an efficient extension to RTDS in simulating large-sized AC networks without losing model accuracy and with no interface errors, while the configuration of the system and results visualization are conveniently carried out using RTDS.
This paper is organized as follows.Section II introduces the EMT models of power system components and the modelling of controllers for synchronous machines.Section III introduces the architecture of the co-simulator.The design of interface between FPGA and RTDS is discussed.Section IV explains the hardware implementation of the AC system.Discussion on the expandability of the model in FPGA is presented.Section V shows the results of two simulation case studies to verify the accuracy and simulation capability of the proposed co-simulator.Section VI discusses the relationship between the size of network to be simulated and the hardware resource utilization & minimum simulation time-step.Finally Section VII concludes the paper.

II. EMT MODELS OF POWER SYSTEM COMPONENTS & CONTROLLERS
To correctly simulate the dynamic behavior of AC power systems, it is essential that both the power system components and the control systems for synchronous machines are modelled in detail.

A. POWER SYSTEM COMPONENTS 1) SYNCHRONOUS GENERATOR
The Universal Machine (UM) model [26] for synchronous generator is adopted.It is a generalized machine model that can be used to represent different types of rotating machines for EMT studies.One d-axis damper winding and two q-axis damper windings are modelled for each machine.The dynamic equations describing the electrical and mechanical parts of the synchronous generator are shown in equation ( 1)-(3): where v dq0 , i dq0 , λ dq0 and u are vectors of stator dq0 voltage and excitation voltage, dq0 winding currents, dq0 flux linkages and speed voltage terms due to the dq0 transformation.R is the resistance matrix consisting of resistance values of stator, rotor and damper windings.L is a symmetrical matrix consisting of self, mutual and leakage inductances of the machine.T m , T e , ω r , ω base , D, H are mechanical torque, electrical torque, rotor angular frequency, rated angular frequency, damping torque constant and per unit inertia of rotor.All variables in (1) to (3) are in per unit except time, which is in second.The positive direction of stator current is flowing out of the machine.
To facilitate model implementation using FPGA, (1)-( 3) are discretized according to the rule of trapezoidal integration and is realized by 6 sub-modules in the actual FPGA hardware implementation as will be shown in yellow blocks in Fig. 6 and Fig. 13.It should be mentioned that the calculations of v dq0 in step 3 and i G in step 5 share the same hardware module as they are both performing the transformation between abc and dq0 reference frames.The descriptions of these sub-modules and the parallelism between them are provided in Section IV.

2) TRANSMISSION LINE
The distributed parameter transmission line model is adopted, which is solved in modal domain before converting back to phase domain.The hardware implementation of the model consists of 2 submodules: phase/modal transformation module and transmission line solver.The transformation module converts all relevant voltages and currents from phase domain to modal domain for calculation, and converts the results from modal domain back to phase domain after solution.

3) PASSIVE ELEMENTS
Trapezoidal rule of integration is utilized to discretize the dynamic equations for inductors and capacitors.As a result, all system inductors and capacitors are represented as the parallel connection of history current terms and constant impedance [24].The history currents are directly used by the ''Nodal Equation Solver'' described later to calculate the system nodal voltages.

4) FAULTS
Fault changes the network admittance matrix and system equivalent resistance.In the proposed EMT co-simulator, these matrices are pre-stored and are chosen according to the fault scenarios.As discussed in previous section, most of the fault cases are implemented using RTDS taking advantage of the flexibility of its GUI.

B. CONTROL SYSTEMS FOR SYNCHRONOUS MACHINES 1) EXCITATION SYSTEM
For demonstration purpose, IEEE Type AC1A excitation system [27] is modelled to control the terminal voltage of synchronous generator.The block diagram representation of excitation system is shown in Fig. 1 where u 1 , u 2 , v f , i f are reference voltage, measured terminal voltage, field excitation voltage and field excitation current.The rest are internal variables to the excitation system itself.Once G1-G8 are discretized according to the trapezoidal rule of integration, the FPGA module computes the value of v f for next timestep using the calculated terminal voltages of v d and v q from this time-step.

2) GOVERNOR/TURBINE SYSTEM
For the purpose of demonstration, the governor/turbine system as shown in Fig. 2 is simulated using FPGA.u 1 is the speed reference, u 2 is the measured rotor speed, u 3 is the load set point, x 1 to x 4 are internal variables and T m represents the mechanical torque in per unit.1/R represents the percentage of droop, T G is the time constant of governor and T ch is the time constant of main inlet volumes and steam chest.Similar to the excitation system, the complete governor/turbine system is discretized for implementation in FPGA.The mechanical torque for synchronous machine at next time-step is calculated using the rotor speed, value of x 3 and value of mechanical torque from last time-step.

C. NODAL EQUATIONS SOLVER
Synchronous machines are solved separately from the linear parts of the system due to its nonlinearity.Compensation method [25] is used to provide the interface between them.It calculates the Thevenin equivalent of the linear part of the system and solves the machine terminal current.The machine is then represented as a current source for the solution of whole network using the formulated nodal voltage equations.3) user PC.The FPGA and RTDS are carrying the main computational burdens, while the user PC is used for results visualization and system manipulations.

A. INTERFACE DESIGN 1) STUDY SYSTEM & EXTERNAL SYSTEM
The study of large-scale power system can be carried out by dividing the system into two subsystems: the study system and external system.The study system is the part of system that is of research interest, where frequent modifications of controller parameters, system topologies and fault conditions are normally required.The external system is the rest part of the system external to the study system, which is normally of much larger size but requires much less modifications during simulation studies.
For the proposed FPGA-RTDS co-simulator, the study system is simulated in RTDS, taking the advantages of the RSCAD GUI for system modification.The external system is simulated in FPGA, making good use of the inherent paralleled architecture and high computational power of FPGA.

2) INTERFACE IMPLEMENTATION
The study system and external systems are interconnected through one or more transmission lines.In this way, the subsystems at both ends of the transmission lines are computationally decoupled, as long as the wave travelling time is equal to or longer than one simulation time-step.
Fig. 4(a) shows the mathematical representation of the interface between FPGA and RTDS, where v RTDS , i RTDS, v FPGA and i FPGA are the terminal voltages and line currents at RTDS side and FPGA side respectively; I h_FPGA and I h_RTDS are the history currents; τ is the wave travelling time.The interface transmission line is represented by two Norton equivalent circuits, one for each terminal.The history currents of the Norton equivalents depend on the voltages and line currents from both ends, i.e., Therefore from the point of view of RTDS, only the terminal voltage and line current of FPGA end at time t-τ are required for the solution of complete system.Similarly, only the terminal voltage and line current at RTDS end are required for solutions at FPGA side.The computations for both I h_FPGA and I h_RTDS can be carried out in either FPGA or RTDS, and the calculated results need to be sent to the other end.
Alternatively they can be calculated in a distributed manner at their respective ends.Fig. 4(b) shows the physical representation of the interface between FPGA and RTDS for the proposed method.As can be seen from the figure, the SFP transceiver on FPGA board is connected by bi-directional optical fiber to the I/O fiber port on RTDS PB5 processor card.It is used to exchange the terminal voltages and currents between both ends.The synchronization signal from RTDS and signals from FPGA that need to be visualized are also sent through the fiber.The fiber used in the current setup is an OM1 62.5 LC-LC multimode cable with a baud rate of 2.0 Gigabaud with 10b/8b encoding.It is able to bi-directionally transfer up to 64 32-bit signals in each direction in every simulation time-step.There are up to 4 I/O fiber ports available on each PB5 card.The Aurora protocol developed by Xilinx is used for high speed point to point communication.
It is important to mention that the implementation of Norton equivalent at RTDS side does not require access to the low level codes of RTDS, as its impedance is purely resistive and no ''history terms'' are required.The history current and equivalent parallel impedance of Norton equivalent can be modelled using controlled current sources and resistors.
At the FPGA side, computations are synchronized with the RTDS at the start of each simulation time-step using the synchronization signal.The actual network solution, which is decoupled from that at RTDS side, is carried out once the exchange of data between the FPGA and RTDS is completed.The calculated terminal voltages and currents are sent to the communication hardware module, as shown in Fig. 5, waiting for the exchange of signals with RTDS side.

3) OUTPUT FROM FPGA
Detailed simulation results of the external system are computed and stored in FPGA.They can be exported in various ways.For example, data can be exported to PC through USB cable or RS232.It can also be displayed on an oscilloscope through a D/A converter.For the proposed co-simulator, FPGA output signals are exported to RTDS through the fiber connection between FPGA and PB5 card as shown in Fig. 4(b) and visualized using RSCAD.On one hand, the GUI of RSCAD can be used for better manipulation of the output signals.On the other hand, it would be more convenient when simulation results from FPGA and RTDS both be accessed through RSCAD.

IV. HARDWARE IMPLEMENTATION
For the hardware implementation using FPGA, the following hardware modules have been designed for most of the commonly used power system components: passive elements module, transmission line module, nodal equations solver module, synchronous machine module and the excitation system module & governor/turbine system module for synchronous machines.The interconnections between these modules are illustrated in Fig. 5.It can be seen form the figure that each module is made responsible for the calculation of one type of power system/control element to take advantage of the inherent parallelism between different hardware modules.
At the same time, deeply pipelined calculation algorithms have been designed within each module itself.So the calculations of all the elements of the same type that are sharing the same hardware module are carried out in a pipelined manner.In some cases the hardware module may need to be duplicated for one particular type of element when the number of elements is large or the computation is complex.This way of design effectively avoids the potential problem of computation delay caused by one excessively complex module.

A. PARALLELED COMPUTATION SCHEMES IN FPGA
Considering the inherent parallelism between modules and the pipelined computation within each module, 7 steps of calculations are designed within each time-step as shown in Fig. 6.It can be seen that more than one hardware modules are processed in parallel in most of the steps.
Step 0: The terminal voltages and currents of the interface transmission line are exchanged between FPGA and RTDS.
Step 1: The calculation of history currents of passive elements and transmission lines and the prediction of rotor speed and angle are carried out in parallel.Step 2: Nodal current injections (I node ) and the park transformation matrix are calculated in parallel using the history currents and the predicted rotor angle from step 1.
Step 3: Using the nodal current injections from step 2, the equivalent system open-circuit voltage v o for each synchronous machine and its value in dq0 reference frame, i.e. v dq0 are calculated.At the same time, the inverse of park transformation matrix and the excitation voltage v f for each synchronous generator are calculated in parallel.
Step 4: This step is to calculate the stator currents in dq0 reference frame, i.e. i dq0 for each synchronous machine.The values of v dq0 , v f , and predicted rotor speed ω p from previous time-steps are required as inputs.
Step 5: The calculated i dq0 from step 4 is taken as input for the calculation of flux linkages (λ), and stator current in abc reference frame (i G ) for each synchronous machine.Then the system nodal voltages v node are calculated using the calculated values of i G .
Step 6: The currents of passive elements and transmission lines are calculated in this step.Also the rotor speed of each synchronous machine is calculated using the flux linkage and mechanical torque input from the governor system.These values will be used in step 1 of next time-step.
Upon completion of step 6, the simulator will idle until the end of current time-step before starting the next step 0.

B. PLATFORM EXPANDABILITY
The proposed co-simulator has been designed so that the simulated AC system in FPGA can be conveniently expanded without significant sacrifice of hardware resources and computational efficiency.Therefore each of the hardware modules shown in Fig. 5 has been designed as a fully self-contained module with dedicated interface signals for VOLUME 6, 2018 interconnection with other modules.In addition, the pipelined computation schemes of elements in each module greatly facilitate the system expansion.Fig. 7 illustrates how system expansion can be carried out with the modular design.Take transmission line module as an example, Fig. 7(a) schematically illustrates how the additional components can be accommodated in the designed hardware modules.In the figure, TL1-TLN are N transmission line components that have been simulated in the original system, and suppose that additional k transmission line components need to be added.The computations of these additional transmission lines can be pipelined with the existing ones so the increase of computational time of this hardware module is k FPGA clock cycles.Due to the high clock frequency of FPGA, the corresponding increase of time is very small.Similar methods are applied to other modules.At the same time the system admittance matrix is expanded as shown in Fig. 7(b).Due to the decoupling effect of transmission lines, the expanded part of the system (with a system matrix of YN+1) can be solved independently from and in parallel with existing networks.Another important aspect with system expansion is its effect on minimum simulation time-step.The proposed expansion method can effectively reduce the increase of the minimum simulation time-step.If the method of pipelining to accommodate more components does not lead to longer solution time than the hardware modules that are processed in parallel, its impact on the overall simulation time-step is minimal.On the contrary, if the network expansion is significant and results in an excessive amount of one particular type of component, additional hardware modules may need to be added.A more detailed discussion on this aspect will be provided in the next section based on the case studies.

V. SIMULATION RESULTS
Two case studies using the proposed co-simulator based on Xilinx Virtex-6 FPGA and RTDS are presented in this section.Case 1 is to verify the accuracy of the proposed co-simulator and case 2 is to demonstrate the simulation capability of the co-simulator.The achieved minimum simulation time-steps are 5.71 µs for case 1 and 20.16 µs for case 2 with 100 MHz FPGA clock frequency.

A. CASE 1
Fig. 8 illustrates the simulation case.It can be seen in Fig. 8(a) that the two-area four-machine system is divided into two subsystems when simulated using the proposed co-simulator.The area with G1 and G2 are simulated using RTDS, while the other area with G3 and G4 are simulated in FPGA.The two subsystems are interfaced by the transmission line TL78.The system parameters are obtained from [28].The excitation and governor systems for the generators are as described in Section II.To verify the accuracy of the co-simulator, the same system is simulated in RTDS as a comparison, which is illustrated in Fig. 8(b).A 150ms three-phase fault located at bus 7 is simulated.Fig. 9 shows the comparisons of simulation results.Fig. 9(a) and Fig. 9(b) compare the simulation results of three-phase voltages at bus 6 from the co-simulator and RTDS.It can be seen that once fault happens, the voltages are dropped with high-frequency oscillations.Detailed agreement can be observed.Fig. 9(c) shows the comparison of electrical torques of generator G4.It can be seen that the oscillatory behaviours are accurately simulated using the proposed method.Fig. 9(d) to Fig. 9(g) show the comparison of transmission line currents on TL78a and TL89a.It can be observed that both the transients and the unbalance between three-phases are accurately simulated using the proposed co-simulator.
The matching of simulation results between the proposed method and the RTDS is expected as the same type of component models (EMT models) are used for both the FPGA and RTDS sides.Therefore no special interface technique (required for EMT-TS hybrid simulation) needs to be designed.At the same time, the size of the network that can be simulated is not compromised which will be illustrate in case 2. In addition, to further demonstrate the simulation potential of the proposed method, the relationships between the size of the network and the minimum simulation timestep/hardware resource are discussed in Section VI.

B. CASE 2
To demonstrate the capability of simulating large AC networks using the proposed co-simulator, a system with 141 three-phase buses are developed as shown in Fig. 10.Five buses and two machines (G37 and G38) are simulated in RTDS, and the rest of the network is simulated in FPGA.The two subsystems are interfaced by transmission lines TL3A and TL3B.The simulated fault is a 100 ms three-phase fault located at bus 141.
As a comparison, if the same system is simulated using only RTDS, 4 PB5 racks will be needed, while with the proposed method, only 1 FPGA board and half of the computational resource in 1 PB5 rack are used.Furthermore, as will be shown in Section VI, the remaining hardware resource of FPGA board is still significant.This means that even larger systems can be simulated based on the current setup without hardware expansion.
Fig. 11 and Fig. 12 show the simulation results using the proposed co-simulator.It can be seen that once fault happens at bus 141, the voltages at bus 140 at RTDS side (Fig. 11(a)) and bus 41 at FPGA side (Fig. 12

VI. DISCUSSIONS
To examine the relationship between hardware utilization and the size of AC network, Table I shows the FPGA resource utilizations for case 1 and case 2. It can be seen from the table that the utilization rate of each type of hardware resource is only increased by a factor of 3.3 -3.7, while the size of FPGA part of the system is increased by more than 20 times.It is mainly due to the expansion method as discussed in Section IV.
To analyze how network expansion affects the minimum simulation time-step, Fig. 13 shows how the minimum simulation time-step is affected by the processing time of paralleled hardware modules.Taking Case 2 as an example, the detailed timing for each step is shown in the figure when   the FPGA clock frequency is 100 MHz.The horizontal axis represents the minimum simulation time-step of FPGA, which can be further divided into step 0-6 as discussed in Section IV.As can be seen from the figure, the processing time of each step is determined by the most time-consuming hardware module in that step.For example, the processing time of step 1 is determined by the transmission line module.Therefore, the increase of calculation time to other modules in the same step does not lead to an increase of the minimum simulation time-step, as long as it is still smaller than the largest one.For example, at step 1 of case 2, the processing times of passive element module, transmission line module When a large numbers of synchronous generators need to be added, the processing time of all steps are affected.Additional hardware modules are needed to increase the level of parallelism to maintain the simulation time-step.However considering the achieved minimum simulation timesteps of 5.71 µs for case 1 and 20.16 µs for case 2, the size of network expansion that can be accommodated would be significant before the minimum simulation time-step reaches the typical RTS time-step of around 50 µs.

VII. CONCLUSION
This paper has proposed an FPGA-RTDS co-simulator for the real-time simulation of large power systems.Under this arrangement, the FPGA provides an efficient extension to the RTDS, alleviating the limitations on the size of AC system that can be simulated with detailed EMT models.At the same time, it avoids the interface errors existing in hybrid EMT-TS simulation methods.To achieve better expandability of the co-simulator, hardware modules have been designed as fully self-contained modular components for most of the commonly used power system elements.They can be conveniently duplicated when the size of the system expands, and the effect on simulation time-step is minimized.Two case studies have been presented to verify the accuracy and capability of the proposed co-simulator.Discussions based on the case studies have been presented to further demonstrate the potential of the proposed co-simulator in simulating even larger systems with sufficiently small time-steps.

Fig. 3 FIGURE 3 .
Fig. 3 shows the architecture of the proposed FPGA-RTDS co-simulator.It can be seen from the figure that it consists of three main parts: 1) FPGA boards; 2) RTDS racks and

FIGURE 6 .
FIGURE 6. Computation schemes of subsystem in FPGA.

FIGURE 7 .
FIGURE 7. Hardware implementation of network expansion (a) Network expansion of transmission line components; (b) Network expansion of admittance matrix.

FIGURE 8 .
FIGURE 8. Simulation of two-area four-machine system.(a) Using the proposed co-simulator; (b) Using RTDS only.

FIGURE 9 .
FIGURE 9. Simulation results for case 1.(a) Bus 6 voltages from RTDS.(b) Bus 6 voltages from the co-simulator.(c) Electrical torques of G4 from RTDS and the co-simulator.(d) Three-phase line currents on TL78a from RTDS.(e) Three-phase line currents on TL78a from the co-simulator.(f) Three-phase line currents on TL89a from RTDS.(g) Three-phase line currents on TL89a from the co-simulator.
(a)) are decreased.The three-phase currents on the connected transmission lines, i.e., TL2 at RTDS side and the line between bus 42 and bus 41 at FPGA side are increased as shown in Fig. 11(b) and Fig. 12(b).The electrical torques of G37 and G38 at RTDS side are shown in Fig. 11(c).Higher oscillations are observed in electrical torques of G38 as it is located electrically closer to the fault.Similar oscillatory responses of the stator d-axis current of G37 and G38 are shown in Fig. 11(d).The electrical torques of G14 and G15 are shown in Fig. 12(c) where smaller oscillations can be seen.

FIGURE 10 .
FIGURE 10.Single-line diagram of the 141-bus system.

FIGURE 13 .
FIGURE 13.Breakdown of minimum simulation time-step of Case 2.