Self-organising technique for carrier synchronisation and phase offset distribution in modular, fault-tolerant converters

: Carrier interleaved pulse width modulated modular multilevel converters provide multiple benefits including increased effective switching frequency, switching frequency harmonic elimination, and reduced filter requirements. For converters comprising high numbers of switching modules, control is often distributed across multiple controllers, making carrier-phase synchronisation challenging. This paper presents a technique for carrier-phase synchronisation and phase offset distribution for controllers in a ring network comprising duplex serial links. The proposed method is self-organising, with equal phase offset automatically achieved across connected nodes. A key advantage of the proposed technique is capacity to enable fault tolerance, through automatic phase offset redistribution in the case of switching modules being removed from the system. An experimental prototype is implemented and evaluated using three FPGA nodes connected via 6.25 Gbps full-duplex fibre-optic links, with phase offset accuracy to within 0.1 degrees demonstrated for 25 kHz pulse width modulation.


Centralised versus distributed control for modular converters
In high-power applications for which the current and/or voltage rating of individual semiconductors limits the overall system performance, the use of multiple sub-rated converter modules is an attractive alternative. Conventionally, a single, centralised master controller is responsible for the control of every converter module in the system, however, this approach does not scale well due to the overhead associated with communicating the system state to the master and individual control signals to every converter module. From a reliability perspective, this also results in a single point of failure for the system: the master controller [1]. Furthermore, as the size of the system and communication demands increase, the probability of failure of a component of the system also increases.
An alternative approach is to distribute the control functions among a number of local control nodes. This reduces the communication burden, as local measurements can be applied to local, or neighbour-based control, rather than being required to propagate throughout the entire system. Reliability is also enhanced, as healthy sub-modules can continue to function in the presence of one or more module failures, eliminating the single point of failure, and reducing the probability of total system failure.
Fault tolerance has previously been explored in converters from two main approaches. Redundancy has been built into systems with extra sub-modules running below capacity [2][3][4][5], or the control algorithm has been revised on the fly to compensate for the loss of modules [6,7]. The main objective to these approaches is to maintain system operation (potentially at a reduced output power level), while additionally maintaining balance among the converter modules and low-order harmonic cancellation.

Synchronisation of distributed controllers
When multiple converter modules are utilised, additional harmonic cancellation benefits can be achieved through the multilevel modulation strategy of interleaving, in which the carrier phase offsets of the individual modules are equally distributed over one switching period. In the case of a centralised controller, the process of achieving carrier synchronisation and phase offset distribution is trivial. This is not the case for a distributed system, however, in which each of the local nodes typically maintains a clock domain which is asynchronous to all other nodes, and which communicates via links with variable latency.
A high level of synchronisation precision is required to achieve good harmonic cancellation in a modular converter, even for low sub-module counts [8]. As the sub-module count increases, the precision necessary to achieve good cancellation of the higher order switching frequency harmonics also increases. Consequently, a system that has the ability to evenly distribute the carrier waveforms with minimal communications demand, and which is modular, scalable and robust is highly desirable. In the faulttolerance context, it is also desirable that the system can dynamically adjust to the addition or removal of sub-modules.
A range of synchronising methodologies for modular converters have been previously presented, including systems based on bus [1,9], simplex ring [10], and full-duplex ring [11] communications topologies, as well as communications-free designs [12]. Performance and features vary greatly across these systems, with no single design combining scalability, self-organisation, and dynamic reconfiguration, which are desirable for operating a faulttolerant modular converter.

Proposed carrier synchronisation and phase offset methodology
In this work, we present a carrier synchronisation and phase offset methodology aimed at enabling the development of fault-tolerant, distributed controllers for modular converters. In this section, we describe the proposed system topology to which this methodology is targeted, including relevant design features. Following this, the proposed carrier synchronisation and phase offset mechanism is identified and explained. Fig. 1 shows the distributed control and communications topology proposed in this work for modular converters using multilevel modulation strategies. The design features a number of distributed control nodes connected in a ring network via full-duplex serial links. These links would typically be realised as multi-gigabit fibre-optic links, which provide high-throughput (up to 10 Gbps line rates), low-latency, and excellent immunity to noisy electrical environments. A system-level controller is responsible for providing common control references to all of the distributed control nodes. Where redundancy is desired, this system level controller could be duplicated. Communications redundancy is achieved by providing multiple connections between the ring network and the system controller, providing multiple possible paths for signal propagation in the case of failure of one or more communications links. Similarly, the full-duplex ring network provides redundancy, as sampled values and control signals can propagate around the network in either direction.

Distributed controller topology
In this model, we assume that each distributed control node operates in its own clock domain, asynchronous to all other nodes (albeit at the same nominal clock frequency). It is also assumed that each node controls the pulse width modulation (PWM) of one or more converter modules, and that it is desirable for the carriers of the modulators to be equally phase distributed across one carrier period.
The remainder of this paper focuses on the presentation of a methodology to achieve equal carrier phase offset distribution in a distributed controller matching the model proposed in Fig. 1.

Carrier phase synchronisation and offset
It is desirable to develop a synchronisation and phase offset methodology for the distributed controller model proposed in Fig. 1, which is tolerant to the dynamic reconfiguration of the system. This characteristic would support fault-tolerance by permitting modules to be removed from the system in the case of failures. As modules are removed, the phase offset distribution must be adjusted to maintain optimum harmonic cancellation.
In this work, we propose a self-organising synchronisation and phase offset distribution technique, based on synchronising signals communicated from neighbouring nodes over the full-duplex serial links. The self-organising nature of the methodology has the additional benefit of permitting flexible system reconfiguration via appropriate connection of inter-node communications links.
The proposed synchronisation methodology is summarised graphically in Fig. 2. Each node in the system maintains a local carrier signal used to synthesise the PWM control signals for its local converter module(s). At the zero-phase instant, each node sends a synchronising signal to its two neighbouring nodes via the communications links. Similarly, each node will receive a synchronising signal from each of its two neighbours. On receipt of these synchronising signals, the node can register the value of its local carrier.
In a system where the phase offsets are equally distributed across one carrier period, the zero phase time instant for any given node should be equally spaced between those of its two neighbours. Each node can, therefore, measure its phase error with respect to its neighbours as per (1): where T pd,comms is a fixed propagation delay between the zero phase signal of a node and receipt of this signal in a neighbouring node, including all communications latency. The phase offset error, T error , can then be applied as the input signal to any conventional PLL implementation to achieve phase locking of the PWM carriers across the distributed control system.

Experimental validation
As a proof of concept demonstration of the proposed carrier synchronisation and phase offset distribution technique, a hardware prototype comprising three nodes connected in a ring network via fibre-optic links was developed; this section documents the experimental implementation, with results and discussion provided in Section 4.

Hardware prototype
The hardware prototype comprises three FPGA modules featuring a Xilinx Zynq-7000 series FPGA SoC and four SFP+ multi-gigabit transceiver interfaces, supporting line rates up to 6.6 Gbps. The FPGA modules are connected via a high-speed full-duplex serial link over optical fibre using the Aurora 8b/10b protocol operating at 6.25 Gbps. Each link between modules is via 1 m of OM3 multimode fibre-optic cable, with a 10 m cable available to test sensitivity to propagation delays in the serial links. Fig. 3 shows a photograph of the experimental prototype.

Synchronisation module implementation
The synchronisation module is based on a phase accumulated carrier pulse width modulator (PACPWM) [13], which uses a 32bit accumulator and 100 MHz clock to synthesise the carrier signal. Frequency is controlled via a 32-bit phase increment word, providing very high average frequency precision.
On the master node, the phase increment word is fixed to produce a nominal reference frequency of 25 kHz, while on the slave nodes, the phase increment is controlled by the output of a phase locked loop (PLL).
The overflow of the PACPWM counter triggers a two-word packet containing dummy data to be sent over the fibre-optic links to both neighbouring nodes. On receipt of the corresponding synchronising signals from its neighbours, the value of the PACPWM accumulator is registered. When treating the accumulator as a signed integer, this provides a direct representation of T pd,+ and T pd-from which the phase offset error can be calculated. A fixed value for T pd,comms was determined experimentally based on observed propagation delays. The phase offset error T error , as defined in (1), was calculated following every pair of synchronising signals received, and passed to the PLL. The PLL was implemented with a proportional-integral (PI) loop filter with gains of 6.1 × 10 −5 and 1.5 × 10 −5 , respectively.
The top bit of the PACPWM accumulator was routed to one of the general purpose digital outputs available on the FPGA module, essentially producing a 50% duty cycle square wave synchronous to the carrier, in order to measure and evaluate the resulting phase offset between nodes.

Methodology
Phase offset between nodes was measured using both a Tektronix DPO3054 oscilloscope (500 MHz, 2.5 Gsps) and a Picoscope 3406DMSO eight digital channel oscilloscope (500 Msps). Phase offsets were assessed in terms of precision and variability.
The sensitivity of the system to communications link propagation delays was assessed by substituting one of the 1m fibre optic cables for a 10 m cable, introducing ∼45 ns of additional latency into the link.
The phase locking dynamics of the system were also analysed, with the phase error observed and evaluated over multiple PWM cycles following the enabling of the synchronising mechanism. Fig. 4 shows an oscilloscope capture of the prototype system in steady state, demonstrating successful synchronisation and phase offset across the three nodes. Excellent phase offset accuracy and precision is observed, with a mean delays between the master and two slaves of 13.34 and 26.67 μs, with standard deviation below 5 ns. Considering the measured delay between the master and slave 0 (see Fig. 4), the majority of observed edges fall within a 20 ns window, with all observed edges falling within a 40 ns window. As the system clock frequency is 100 MHz, this represents a maximum deviation of ± 2 clock cycles. While there remains a small margin to further improve the phase synchronisation (up to the limit of ± 0.5 clock cycles), the observed performance would be sufficient in most practical applications.

Synchronisation precision and phase dynamics
The dynamics of the phase locking process have also been recorded for the two slave nodes, with the master used as the zerophase reference. These data are presented in Fig. 5 for an arbitrary initial system state, with the synchronisation mechanism enabled at time zero. Plotted in Fig. 5b are the phase errors for the two slaves, with respect to the target phase offsets of ± 120°. Note that the dynamics of the synchronisation process are appropriately fast, with the phase error having settled to within ± 10° in 1 ms (25 PWM cycles) and to steady state within 2 ms (50 PWM cycles). Further work is required to investigate fine tuning of the PLL PI loop filter and assess the effect of the gains on phase offset precision and variance.

Self-organisation
To demonstrate the self-organising feature of the proposed methodology, the three-node system was reduced to a two-node system. In this configuration, the appropriate phase offset for the single slave would be 180°. Fig. 6 shows that the correct phase offset was successfully achieved for the two-node system, with dynamics comparable to that of the three-node system.

Sensitivity to communications propagation delay
The sensitivity of the system to communications propagation delay was investigated by substituting a 10 m fibre-optic cable for one of the 1-m fibre optic cables in the prototype system. The additional length of the cable introduced an approximate 45 ns additional delay in the corresponding communications link. Two tests were conducted: introducing additional propagation delay between the master and slave 0, and introducing additional propagation delay between the two slaves. The results are summarised in Fig. 7.
As expected, altering the propagation delay between nodes introduces a phase offset error, as there is a fixed compensation in the prototype system for propagation delay. This is clearly seen when comparing the steady-state phase offset errors in Fig. 7a (the control case), with Figs. 7b and c (in which additional propagation delay is introduced). As the master is used as the zero-phase reference, the effect is slightly different in the two cases, resulting from how the additional delay is shared between nodes. In the symmetrical case, where the delay is introduced between the slaves and equidistant from the master, it can be seen that the phase errors are equivalent across the slaves (see Fig. 7c).

Limitations
During experimental validation of the synchronising system, several limitations of the proposed methodology were exposed and unexpected behaviours observed. First, it was observed that ambiguous phase ordering resulted, depending on the initial state of the system. This behaviour is demonstrated in Fig. 8a, with slave 0 phase locking to the −120° phase offset, while slave 1 phase locks to 120°; this ordering is reversed compared with the results presented in Fig. 5a. Similarly, instances were observed in which all nodes synchronised with zero phase offset, as demonstrated in Fig. 8b.
This result can be attributed to the use of an error signal for the PLL which is symmetrical, regardless of the phase offset ordering of the neighbouring nodes, and which considers only the relative difference in delays between neighbours, rather than absolute phase delays. It is expected that the ambiguity in steady-state phase offset would be exacerbated by increasing the number of nodes, as this would provide significantly more combinations of stable system states which produce non-equal phase offsets between neighbours (and potentially different carrier frequencies).
The sensitivity of steady-state phase offset to initial conditions limits the usefulness of this prototype implementation. It is expected that this shortcoming could be eliminated through the inclusion of additional logic to enforce positive, and non-zero phase offsets between nodes in a specific direction around the ring network. For larger systems, propagation of information among nodes, such as the total number of nodes in the system, may be necessary to ensure a deterministic steady state is achieved. Table 1 documents the resource utilisation of the FPGA implementation used in the prototype system. The synchronising methodology developed in this work, composed primarily of the PLL and PACPWM modules, consumes a negligible amount of the overall FPGA resources; ample capacity is available for extension and refinement of the synchronising system. The utilisation is dominated by the balance of the system, which includes instantiation of the Zynq Processor System (with associated ancillary interfaces), and four Aurora multi-gigabit serial transceivers (two of which are unutilised in this design).

Conclusion
This paper presents a self-organising technique for carrier synchronisation and phase offset distribution in modular   converters. The technique applies to distributed controllers connected in a ring network via full-duplex serial links. A proof of concept experimental validation of the proposed technique has been conducted, with phase offset errors below 0.1° achieved for a three-node system and 25 kHz PWM. While precise synchronisation was observed, further refinements to the methodology are required to overcome limitations identified in terms of ambiguous phase ordering and incorrect steady-state phase offsets.