Design and implementation of the MMC simulation system in the heterogeneous FPGA-CPU platform

: Real-time simulation of a modular multilevel converter (MMC) plays an important role on the area of large-scale power electronics research. The authors propose a real-time MMC simulation system in a heterogeneous computing platform containing the central processing unit (CPU) and field programmable gate array (FPGA). This system decouples the MMC circuits from the grid based on the alternative circuit equivalent model of the MMC bridge arm. Furthermore, the parallel calculation of the equivalent circuits is achieved when the peripheral circuit is realised on the CPU and the MMC arms are realised on FPGA. In addition, a timing optimisation strategy is discussed to reach the timing target requirement. To validate the MMC simulation system, a case study of a grid with 12 MMC bridges and 640 sub-modules per bridge at 2 μs time step on Xilinx Virtex-7 XC7VX690T FPGA is simulated in real time. The real-time simulation results demonstrate high accuracy of the simulation system in comparison to the offline simulation of the original system in the electromagnetic transient program and our design is competitive to recent published works.


Introduction
Due to the advantages of low harmonic, low switching frequency, high modularisation, and good scalability, MMC is widely used in the middle-and high-voltage power situations [1].The complex circuit topology and control system of modular multilevel converter (MMC) challenge the stability and reliability of the system.The research and simulation of the MMC electromagnetic transient process under the abnormal working conditions become a hot topic of electromagnetic transient simulation.
The real-time simulation of electromagnetic transient can be used in the hardware closed loop experiment to verify the physical performance of the device and the system [2][3][4][5].In order to achieve real-time simulation speed, it is required that the electromagnetic transient simulation system has a very small calculation step.The server based on CPU architecture can provide many computing resources.However, the CPU-based serial architecture limits the calculation step in electromagnetic transient simulation based on CPU servers.At the moment, relying on the reconfigurable and highly parallelised architecture, field programmable gate array (FPGA) is gradually applied to many electromagnetic transient real-time simulation research.It is widely used in the closed-loop examination of external hardware such as controller and relay protection, and has become a powerful tool for power system simulation [6,7].However, the limited resources of FPGA and its expensive cost have become the bottleneck of largescale electromagnetic transient simulation research [8].
In order to fully utilise the computational efficiency of FPGA and reduce the cost of research, the research on MMC arm simulation based on the FPGA-CPU platform has been carried out.Venkata Dinavahi of University of Alberta in Canada studies the power consumption model of the MMC circuit, and implements the MMC model simulation system based on FPGA, but it only reaches nine-level output at most [9].The real-time digital simulator (RTDS) company achieves a simulation of 512 submodules of the MMC model based on FPGA; however, FPGA only realises the simulation of three MMC bridge arms, while the actual MMC system contains six bridge arms, and each bridge arm contains hundreds of sub-modules [10].A real-time simulation system of MMC based on CPU and FPGA is put forward [11], and the real-time calculation of the simulation can achieve the scale of six bridge arms of the 400 sub-modules, but the system only simulates the fully operational MMC bridge circuit under normal condition, unable for locking or other failure of the MMC bridge module features simulation.An RTDS-based MMC module equivalent model is proposed [12], simulating single MMC submodule characteristics of MMC simulation.However, the MMC simulation scale is limited.The biggest supports only seven levels, so it is difficult to meet the practical needs of simulation research.A simulation platform for MMC is proposed that can simulate 512level-scale MMC [13].Its simulation step can reach 2.5 μs.However, there still exists some shortcomings such as the small scale of single board simulation and the lack of the parallel degree in the MMC simulation system.
In order to meet the requirement of electromagnetic transient simulation in a large power grid, this paper implements an FPGA-CPU-based MMC real-time simulation heterogeneous computing platform.According to the different requirements of different modules in the power grid, the power grid is divided into the MMC peripheral circuit subnet and the MMC main circuit subnet.The peripheral circuit subnets with lower computing performance requirement are simulated on the CPU-based server, while the MMC main circuit sub-network with higher computing performance requirement is simulated on FPGA.Through the data interaction between the server and FPGA, parallel FPGA-CPU heterogeneous computing is realised.Meanwhile, fine-grained parallel design is used to optimise the simulation rate of the MMC main circuit subnet in FPGA.
This paper first introduces the modelling principle of the MMC circuit in the power grid, and then puts forward the system structure of the heterogeneous computing platform based on FPGA.Then, after introducing the computation process of the server and FPGA, this paper analyses the detailed calculation process of the MMC arm on FPGA and the timing optimisation strategy of FPGA.At the end of this paper, the performance is evaluated and compared by the experimental results, and the effectiveness and superiority of the proposed design scheme is verified.

Equivalent model of the MMC bridge arm adopting the surrogate circuit
MMC electromagnetic transient simulation requires a precision mathematical model of the MMC bridge arm which is easy to be solved.However, the complex MMC circuit structure results in difficulty in establishing the electromagnetic transient equivalent model of the MMC branch.Therefore, the topology of the MMC circuit and a simplified equivalent arm branch decoupling model are introduced in this chapter, which can be used to realise independent simulation of MMC arm circuits.

MMC network topology
The typical MMC topology is shown in Fig. 1a.Each phase is composed of one upper and one lower bridge arms.The bridge arms are connected by N the same sub-modules (SM) and a bridge arm reactor L. U dc is the DC side voltage of MMC, and L, as a protector for the circuit, can suppress the interphase circulation and fault current under serious fault.
The half-bridge sub-module (half-bridge sub-module, HBSM) is the most widely used in MMC.As shown in Fig. 1b, a halfbridge sub-module consists of two IGBTs, and two anti-parallel diodes and a DC capacitor.T 1 and T 2 rotate in turn when the MMC circuit works.If T 1 is turned on and T 2 is turned off, the output voltage equals U c and the sub-module goes into operation.If T 2 is turned on and T 1 is turned off, the output voltage is 0 and the submodule is locked.

Equivalent MMC model of the bridge arm with a surrogate circuit
The MMC model of the equivalent arm with a surrogate circuit cut out a section of inductance L t from the current limiting reactor on the bridge arm as an equivalent transmission line, so that the MMC circuit is decoupled with the whole circuit.Then the MMC bridge arm circuit can easily solve the MMC bridge arm port voltage and port current [8,14] according to the circuit principle.At the same time, the external circuit that does not contain the MMC arm branch can be simply equivalent to a transmission line interface.The usage of the surrogate circuit model decoupling method parallelises the calculation of arm branches and external circuit parts of MMC, greatly improving the efficiency of MMC electromagnetic transient simulation.
The solving process of the equivalent MMC bridge arm model based on the surrogate circuit is divided into two parts: The MMC simulation system designed in this paper is composed of two parts: FPGA and CPU.The MMC arm branch equivalent simulation model (SM) with a surrogate circuit requires high-speed computation, so FPGA is responsible for the simulation of the MMC arm branch.Meanwhile, CPU is responsible for the simulation of the DC side and the AC side of the MMC circuit.

System architecture
The overall architecture of the system is shown in Fig. 2.
As shown in Fig. 2, the subnet of the DC side and AC side outside MMC is mapping to the server based on Linux for simulation, and the equivalent circuit of the MMC arm branch is mapping to FPGA.Meanwhile, FPGA is connected with an MMC controller to control the sub-module trigger signals in MMC circuit models.FPGA is the core of this system, which is responsible for the MMC arm circuit simulation with a large calculation scale and high calculation speed in this system.FPGA is connected with the external MMC controller through optical fibre and interacts by the Aurora protocol.FPGA is also connected with the server through optical fibre.Due to the lack of the optical fibre interface in the server, an expanded peripheral component interconnect express (PCIE)-to-optical fibre card is used in the server to achieve data communication.

MMC peripheral circuit SM based on CPU
The peripheral circuit of the MMC system consists of two parts: the DC side and the AC side.The DC side of MMC is usually a DC voltage source or a DC side port of the other rectifier circuit, while the MMC AC side circuit is connected to the AC power grid or some other AC power supplies.The equivalent transmission line method decouples the peripheral circuit of the simulation MMC system from the MMC arm branch circuit and enables the parallel simulation on the different calculation platforms of the MMC arm circuit and the peripheral circuit.
In this system, the peripheral circuit is composed of a DC voltage source and an AC power grid, whose calculation of electromagnetic transient process is relatively small.Moreover, the economic cost of the electromagnetic transient simulation of the CPU-based platform is relatively low, and the development efficiency is much higher than that of the FPGA-based platform, so the simulation of the MMC peripheral circuit is realised on the CPU platform.The simulation algorithm is based on the typical node analysis method.The electromagnetic equations of the node voltage and the injected node current between the models of the power grid are set up.Node currents are updated according to the electromagnetic characteristics of the power grid internal model, and node voltages are updated through the electromagnetic equations of the voltage and current.The electromagnetic transient simulation calculation process adopts the discrete time step iterative calculation to simulate the change of the electromagnetic transient process.The calculation steps within each time step are composed of three steps.
The first step is updating the node injected current for the corresponding component model in the simulation example.The node voltage and state parameters of the nodes are inputted into the element model, and then the injected current affection of the element model to the grid is solved according to the calculation equations of the corresponding electromagnetic constraints.The injected current of each elements is updated in turn.The injected current of each components can be considered as an injection vector in accordance with the node number.Therefore, through summing updated injected current vectors of all components by linear, the injection current total vector in this time step is updated.
The second step is updating the admittance matrix of the simulated circuit.The admittance matrix is related to the topology of the circuit, which is mainly influenced by the switching action.
So the switch model will be detected before updating the matrix.If the switch action occurs, the updated circuit admittance matrix is selected according to the new switch combination after the action.
The last step is updating the circuit node voltage.According to the new node injected current vector and the circuit admittance matrix, the node voltage value of the next time step is obtained by the matrix solution through the constraints of the node voltage and injected current.
After the iterative computation in the time step, the MMC peripheral circuit model needs to determine whether the simulation is over and whether the data is interacted with the MMC arm circuit model.

Simulation structure of FPGA:
The electromagnetic transient process changes very rapidly in MMC arms.Therefore, in order to achieve high simulation accuracy, the simulation step is required to be as small as possible in the MMC arm simulation.The independent circuit of the MMC arm simulation is implemented on FPGA.The arm calculation and update process mainly consist of four steps as follows: (i) SM status updating, (ii) SM updating, (iii) SM branch merging, and (iv) arm module solving.
An MMC equivalent arm SM simulation structure is shown in Fig. 3.
At the beginning of each step, the system first enables module SM status updating.According to the present states of the SM diode, trigger signals and the fault status, SM status at the next time step is judged by the pipeline for each SM.At the same time, SM capacitance voltages and equivalent resistances are updated according to the parameters calculated at a previous time step.
Then the equivalent voltage sources and resistances of SM branches Thevenin circuit are solved by grouping SMs according to SM states.At the same time, the grid voltage and current of the MMC arm interface are updated.This module will receive the current and voltage from the peripheral circuit, and send the arm interface current and voltage to the peripheral circuit.
After SM merging and arm interface updating, the branch current of the MMC arm is solved in the MMC equivalent branch module.
Finally, the SM state and capacitor voltage are sent to the MMC controller for SM trigger signal updating.
Based on the equivalent model of the MMC arm with a surrogate circuit, the calculation process of the MMC bridge arm is divided into four relatively independent calculation steps as shown in Fig. 4: Step 1: SM voltage and resistance solving Step 2: SM branch merging Step 3: Arm interface voltage and current updating Step 4: Arm equivalent branch current solving Steps 1−4 constitute a complete iterative process of the MMC arm simulation.Besides, FPGA should communicate with the MMC controller for trigger signal updating, and with server for arm interface updating.

System timing analysis and optimisation:
FPGA simulates 640 SMs in every arm.If every SM is updated fully parallel, it will bring heavy consumption of logic resources on FPGA.Therefore, it is necessary to reuse computing resources as much as possible while ensuring system timing constraints.FPGA resource consumption with different parallelism is shown in Table 1.
The FPGA development platform adopts the Virtex-7 VX690T FPGA.As the target frequency of the MMC simulator on FPGA is 150 MHz and the target simulation time step is 2 μs, the clock cycle within each time step should be <300 cycles.As shown in Table 1, it is found that the parallelism requires at least 4 parts parallel of each arm to meet the timing constraints.Meanwhile, the total register resource of the FPGA board is 433,358, so the 8 part parallel simulation of each arm causes the consumed register  resource to exceed the FPGA total resource limit.Considering the resource constraints and timing constraints of the system.This paper divides each arm into four parallel pipelined computing ways to achieve timing optimisation under the premise of system resource constraints.Therefore, 640 SMs each bridge arm is divided into four groups.These four parts of parallel flow calculation is implemented in Step 1 and Step 2. This allows the input delay of Step 1 pipeline to be completely hidden in Step 2. Total timing delay for pipelined data input in Steps 1 and 2 is 160 cycles.The computing delay of Step 1 and 2 is 40 cycles.In addition, Step and Step 2 in each 12 MMC arms are totally parallel.Besides, Steps 3 and 4 of these 12 arms are basically the same, and there is no data dependence between different arms, so this paper adopts the same set of pipelines to calculate step 3 and 4 of every bridge arm in order to save computing resources.This pipeline only produces a pipeline input delay of 12.There is no relationship between Step 3 and Steps 1 and 2. So Step 3 is parallel to Step 1 and 2 sequence, and the calculation delay of Step 3 can be hidden in Step 2. The calculation delay of Step 4 is 48 cycles, and the pipeline input delay is 12 cycles.There is no data dependency and coupling between calculation process of the arm equivalent diode and Steps 1 and 2 at the present time step.So calculating process of arm equivalent diode can be in advance and parallel to Steps 1 and 2 calculation, which will hide the 14 cycles of calculation delay of arm diode module, as shown in Fig. 5.
Through the optimisation and analysis above, the total calculation delay required for each time step on FPGA side is 160 + 17 + 23 + 34 + 12 = 246 cycles.Considering the system clock of 150 MHz, the simulation step of the MMC arm circuit side based on FPGA can be limited within (1/150) 242 = 1.64 μs, which satisfies the design target constraint of 2 μs.

Experiment and result
The FPGA development platform adopts the Virtex-7 VX690T FPGA, and the server adopts the Intel (R) Core (TM) i7-4900 CPU processor.The MMC electromagnetic transient simulation system uses a simulation step of 2 μs.After each iteration of the MMC arm circuit simulation, the communication and data interaction need to be carried out with the peripheral circuit simulation.
The MMC grid structure of the experiment is shown in Fig. 4. The rated voltage of the DC side is 150 kV, the MMC arm rated current is 1.15 kA, the arm equivalent inductance is 48 mH, the SM equivalent capacitance value is 120 μF, and the total amount of SM in each bridge arm is 640.Fig. 6 is the emulation waveform of the normalised bus voltage of the peripheral network AC side entrance at the normal operation of the MMC converter.It shows that the voltage waveform of the MMC output is very close to the sine wave, which is consistent with the theoretical waveform.It proves the correctness of the FPGA-CPU-based MMC real-time simulation heterogeneous platform.Fig. 7 shows the relationship between the simulated time length and the simulating time length.It can be observed that the simulating time is proportional to the simulated time.In addition, the time of the CPU-based simulation is 60 times longer than that of the FPGA-CPU-based heterogeneous platform for the same simulated time length.Besides, the FPGA-CPU-based heterogeneous computing platform uses the same time as the simulated time to complete the simulation task.
As shown in Table 2, compared with the existing design, the system designed in this paper has a higher performance resource consumption ratio.
The simulation results show that the MMC real-time simulation heterogeneous platform based on FPGA-CPU can effectively simulate the power grid containing MMC and meet the real-time requirement of electromagnetic transient computation with a faster simulation speed and at a larger scale.The design of an MMC real-time simulation heterogeneous platform based on an FPGA-CPU-based platform is implemented in this paper.The system can support the electromagnetic transient real-time simulation of the power grid with MMC.In this paper, the research status of the multilevel converter is analysed.To improve the characteristics of the large simulation step and small simulation scale in MMC electromagnetic transient real-time simulators, this paper proposes a design and structure of the FPGA-CPU-based MMC real-time simulation system.The system realises the simulator adopting the surrogate circuit.In addition, the system timing is optimised to satisfy the timing constraint.The real-time experiment results show that this system implements a real-time MMC simulation system with a faster simulation speed and at a larger scale.
(a) The solution of the equivalent transmission line interface circuit of the main circuit (b) The solution of the equivalent model circuit of the bridge arm 3 System design of the MMC simulation system

Fig. 3 Fig. 4
Fig. 3 Structure of MMC arm simulation on FPGA