Digitizer hardware for magnetic data acquisition on COMPASS-U

—This paper presents the design and development of a modular digitizer prototype tailored for magnetic sensor data acquisition within the COMPASS-U tokamak facility, an upcoming experiment in Prague that will operate under conditions relevant to future experiments like ITER and DEMO. The magnetic diagnostic data serves a dual role, both for scientific analysis and real-time control. Due to the latter, the data integrity and fidelity is paramount for the plasma performance and overall safety of the fusion experiment. The digitizer prototype showcases a contemporary adaptation of prior board designs deployed in large fusion experiments such as JET and W7X. Leveraging the advanced capabilities of the Xilinx K26 Zynq Ultrascale+ System-on-Module, this digitizer interfaces seamlessly with ADC modules, facilitating real-time data pre-processing. The digitizer sends data to the real-time control system by a high-speed PCI Express interface. A distinctive facet of the design lies in its configurable clock architecture, allowing versatile distribution of clocks and triggers to accommodate diverse experimental scenarios. The inherent modularity and dynamic reprogramming of the hardware mean the system can find applications beyond its intended role. These include data acquisition for other diagnostics or application in different experiments. The modularity and flexibility and scalability of this design position it as invaluable candidate for high-volume data acquisition systems with a high number of isolated channels at a low cost per channel. As the scientific community endeavors towards advancing fusion technologies, the presented modular digitizer design stands poised to make significant contributions in magnetic diagnostics and real-time control applications


I. INTRODUCTION
Research in magnetic fusion experiments is steadily moving towards longer pulse lengths which places increased requirements on their control, data acquisition and communication systems (CODAC) [1].The largest experiments have a hazardous operation environment due to the high neutron flux caused by the fusion reaction and must be run remotely.
COMPASS Upgrade (COMPASS-U) is a new tokamak experiment (R = 0.894 m, a = 0.27 m, B t = 5 T,I p = 2 MA), under commission in Prague, Czech Republic [2,3].Its scientific goals are relevant to support the operation of ITER and DEMO such as plasma exhaust under extreme heat loads and advanced confinement modes.Additionally the hot wall operation places constraints on the design of the diagnostic systems of the machine [4].
One of the fundamental diagnostics in magnetic confinement fusion experiments is the magnetic diagnostic.A mul-titude of plasma parameters can be measured with magnetic sensors, among them: plasma current, plasma position, plasma shape and magnetic equilibrium.While some of these parameters, such as the plasma position and shape are crucial for the safe and reliable operation of the machine, others like the magnetic equilibrium are essential for other's plasma parametric analysis [5].
In fusion experiments, most of sensors used for the magnetic diagnostic system are inductive sensors.Since they use the principle of magnetic induction to measure magnetic fields, the signal they output is proportional to the time derivative of the magnetic field.As such, to obtain the magnetic field the induction sensor outputs need to be integrated.In the context of magnetic fusion experiments there are a number of integration methods that can be split into two major ones: analog integration and digital integration.Both of these methods are susceptible to integration drift: over time, noise with DC or low frequency components in the input signal accumulate and result in an error in the output of the integrator.As pulse lengths increase this becomes one of the major challenges that must be overcome in order to obtain accurate magnetic field measurements.
The data acquisition (DAQ) system is an integral part of any diagnostic.In the particular case of the magnetic diagnostic, while the bandwidths demanded by plasma shape and position control are quite low (up to the tenths of KHz) the measurement of MHD modes and more advanced instability control algorithms demand higher bandwidths (over 300KHz up to 1MHz in some experiments) [5,6].Combined with the high number of sensors present on large devices, and realtime latency constraints, rapidly puts the demands placed on the data acquisition system outside of those offered by offthe-shelf hardware.This implies the design of custom data acquisition systems tailored to the particular needs of magnetic data acquisition.
Due to the low latency, high data volume constraint of magnetic data acquisition systems most of the DAQ systems feature a Field Programmable Gate Array (FPGA) as its main processing element [7][8][9].Moreover, their highly parallel nature and low latency FPGA's are particularly well suited to the needs of magnetic data acquisition.Recently System on a Chip (SoC) hardware has become increasingly popular.SoC's combine a traditional processor with a programmable logic fabric, providing the benefits of both kinds of hardware.The EPJ Web of Conferences 288, 03005 (2023) https://doi.org/10.1051/epjconf/202328803005ANIMMA 2023 * ease of development and familiar environment of a CPU gets combined with the low latency and highly parallel nature of an FPGA.
In this paper we present the design of the COMPASS-U magnetic data acquisition system.It leverages the capabilities of SoC FPGA devices to implement a flexible hardware design that can be readily reprogrammed to as the needs of the data acquisition system evolve.

II. SYSTEM DESIGN
The design of the data acquisition system as a whole was guided by previous designs already proven to be effective [7,9,10].Since then, these have become obsolete as the electronic components they depend on are no longer manufactured.A major aim of this design is then to harness the capabilities of newer components to satisfy the requirements of the magnetic diagnostics of COMPASS-U at a reduced cost per acquisition channel.A second objective was to create a sufficiently modular architecture that could also be used in other low latency, high channel count and high data volume applications.
The acquisition hardware is part of the overall magnetic diagnostic acquisition system.It stands between the magnetic sensors and the real-time plasma controllers.Its function is to digitize and pre-process the signals detected by the sensors, the acquired signals are then made available to the real-time plasma controllers and are also stored in a central database for offline data analysis.

A. Constraints and requirements
The hardware we developed targets the COMPASS-U magnetic acquisition system and as such, there are a set of constraints and requirements it must meet.Some of these constraints arise from the nature of the magnetic sensors selected for use in COMPASS-U, while others come from the design of the CODAC systems of the machine.The environment of COMPASS-U (up to 500 ºC wall temperature and high vacuum) places several constraints on the design of the magnetic sensors themselves, mainly in the selection of materials for their construction [4].Most of the sensors that will be used in COMPASS-U magnetic diagnostic will be induction coils.Since induction sensors present floating voltage readings they must be digitised with differential input analog to digital converters to prevent common mode errors.Furthermore, it was shown that isolation of the inputs is important to reduce crosstalk and mitigate integrator drift [11,12].In the acquisition system this is done with the modules described in [13], which our hardware is specifically designed to interface.The ADC modules are based on the digital integration with modulation design described in [12], that has been previously used at ISTTOK [14] and W7X [15].
From the design of the COMPASS-U magnetic diagnostic we will need at least 350 synchronous acquisition channels [16], [2].The per channel bandwidth requirements imposed by the magnetic sensors (up to 1 MHz) require a sampling rate of at least 2 MSPS.With a bit depth of 24 bit per sample (the maximum under consideration for COMPASS-U [17]), each channel requires a digital bandwidth of 48 Mb/s which corresponds to a total system bandwidth requirement of 16.8 Gb/s.The CODAC system also places other constraints on the design of the magnetic data acquisition system.The real-time control loop will have a period of 50 µs which constrains the maximum latency on the data acquisition system.There is also the requirement of galvanic isolation between the acquisition hardware and the control hardware.Given the operating environments, bandwidth and latency requirements, this points towards the use of an optical connection for data transfer between the real-time control and data acquisition systems.
A third constraint that was, unfortunately, also present was the low availability of electronic components on major distributors during the beginning of the design process circa 2021.This greatly limited component choice, in particular for the programmable logic components which we found in short supply in major electronic distributors.Since one of the aims of the design was to achieve a low cost per channel, the availability of electronic components on major markets is a major consideration.

B. Acquisition system architecture
The acquisition system we designed is based on the Advanced Telecommunications Computing Architecture (ATCA) standard like previous systems [7,8,15].We chose ATCA, a telecommunications industry standard proven in its reliability and availability archived through the use of redundancy.The form factor allows for high channel densities in a compact chassis.We also considered the mTCA.4 standard since its widely used on other physics experiments [18][19][20], but due to its smaller board size it could not achieve the same channel densities as ATCA.The ATCA standard defines a backplane which a number (from two to sixteen) of boards utilize to communicate with each other through point to point multi Gigabit serial links.The standard offers multiple connection typologies: full mesh, where each board has a link to every board, and dual star topology where two redundant central boards have a link to all the other node boards.Since our design is focused on data acquisition we chose a dual star topology for ease of implementation.
The high level design of the system can be seen in Fig. 1.The system is composed by a central switch board and multiple (up to twelve) node boards.Each node board interfaces up to 24 dual-channel ADC modules, with isolated differential inputs, 18-bit precision and an acquisition rate up to 2 MSPS.With this each board will have 48 input channels which translates in 576 channels per ATCA chassis (assuming a 19" chassis with 14 slots), optical clock and trigger input ports and a set of four SFF-8644 ports for communication with the realtime control system.This communication is established with a PCIe link.The PCIe expansion bus was chosen because it is capable of providing sufficient bandwidth to carry the required data volumes to the real-time control system with low transfer latencies and because it was used successfully on previous data acquisition systems.The SFF-8644 ports can carry a PCIe 3.0 x16 link over optic cables, providing a bandwidth of 16GB/s with sub microsecond latencies and ensuring galvanic isolation between both ends of the connection.
The analog signals go into the the node boards which function as a digitizer.These boards host the ADC modules responsible for digitizing the incoming signals.The modules all interface to an FPGA which gathers the data from the ADC's, numerically integrates the signals in real time and forwards the resulting data to the central board through a low latency, high-speed serial PCIe connection carried over the fabric channel in the ATCA backplane.The central board's primary function is a data aggregator.It has a PCIe switch that is used join the PCIe Gen2 x4 links from each digitizer board into a single x16 Gen 3 up-link to the real-time control system.The central board also has the clock and trigger inputs to the acquisition system which are distributed to the node boards over the ATCA backplane synchronization clock interface.
Both the node and central boards are based on the same design.The central board is a modified version of the node board with the ADC module interface removed and a PCIe switch added.This enables the reuse of most of the board design, considerably lowering the design effort.Both boards contain the same SoM too, reducing the effort of maintaining the software stack.

III. DIGITIZER DESIGN
The design of the digitizer board is centered around the AMD Kria K26 System on a module.This SoM contains a Zynq Ultrascale+ SoC that combines programmable logic (PL), like the fabric of a conventional FPGA, with a multicore ARM processor and a series of peripheral interface hard blocks.Its programmable logic capabilities are well suited to the massively parallel real time computations that a realtime data acquisition system requires.The processor coupled with the peripheral interfaces form a processing system that provides a familiar Linux based programming environment that is useful for system management, remote operation and supervision.The system on a module integrates many of the power regulators and peripherals required for SoC operation such as DRAM, non-volatile storage and firmware storage which considerably reduces development effort.This particular SoM was chosen due to its manufacturer support, availability on major distributors, low cost and high number of I/O pins.The SoM has two types of I/O pins connected to the PL: high performance I/O, which support differential signaling standards at data rates up to 1.25 Gbps and high density I/O pins, which mostly support single ended signaling standards at rates under 250 Mbps.A scheme of the design can be seen in Fig. 2. It illustrates all major subsystems of the design.These are: the ADC interface, the clock and trigger distribution circuits, the high speed serial interfaces and the processor system peripherals.
The use of K26 module as the processing element of the digitizer board brings us some benefits to the overall architecture of the acquisition system.The on-board RAM and storage provide us with and extra layer of redundancy during the data acquisition since its is possible to buffer the acquired data in RAM and later store it on the digitizer board to non volatile storage.This prevents data loss in case of a fault in the data streaming interfaces.
Taking into account the form factor (130 mm x 20 mm x 9.5 mm) of ADC modules and the space available on the board, we have found that 48 channels per board is the maximum density that can be achieved.The ADC modules are mounted on to the main board at a right angle and they are oriented parallel to the airflow on the ATCA crate for temperature homogeneity.Since there are two channels per module there are 24 per board.the signal.This ensures crosstalk between acquisition channels is minimized while maintaining electrical isolation up to 1500 V between each channel.The modules connect to the main board with two right-angle connectors; one of them carries analog signals and the other carries digital signals.The digital connector is a high-density connector for high-speed signals; the connector on the analog size is of a different type and lower pin density, it was selected taking into consideration the electrical isolation requirement.The use of different types of connectors for the ADC module interface adds the obstacle of finding compatible connectors, but allows us to choose the best type for each situation with less compromises.
The digital interface to the ADC modules is based on SPI with a dedicated acquisition trigger signal.The SPI control signals are carried over LVDS to the modules to minimize crosstalk and interference.Since the ADC control signals are the same for every module, they are distributed using fan-out buffers which greatly reduces the demand for I/O pins on the SoC and also simplifies the board layout and routing.The acquisition trigger for the ADC is also distributed to every module using precision clock distribution fan-out buffers that guarantee low jitter on their outputs.To avoid lowering the ENOB of the ADC's we targeted a acquisition clock jitter under 1 ps which is sufficient for 18-bit precision at 1 MHz (The upper limit of COMPASS-U's high speed inductive sensors' bandwidth [16]).Since the mixed-mode clock managers (MMCM) available on the SoC can only obtain a jitter of 200 ps the design features an external PLL to reduce the jitter of the clock signal to acceptable levels.Because the trigger to the ADC is not a symmetric pulse we use a clock and data synchronizer IC to re-time the acquisition trigger pulse coming from the SoC to the low jitter clock synthesized by the PLL.
Each node board can send or receive up to six independent clock or trigger signals from the ATCA backplane.One of the clocks is reserved as the PCIe reference clock leaving the rest of them to be used to synchronize the data acquisition across all node boards.Apart from the PCIe clock which is routed to a PLL to reduce its jitter, all of the clocks are routed to the SoC where they are used to synthesize the acquisition start signal.The clock distribution circuit is shown on Fig. 3.With the use of bi-directional MLVDS buffers and the programmable logic on the SoC we can see that the design is quite flexible and offers several possibilities for clock and trigger distribution to suit the needs of a particular data acquisition system.
The SoC features two sets of four lanes of high-speed serial multi-gigabit transceivers (MGT).One of the sets is controlled by the processor system and is capable of speeds up to 6 Gbps per lane while the other is within the programmable logic section and is capable of speeds of 12.5 Gbps per lane.The processor system MGT lanes are restricted to a small set of communication protocols, notably PCIe Gen2 which we use for data output.The programmable logic MGT lanes allow for a much larger number of communication protocols and as such we opted to have them available on the board's expansion port.The board features an expansion port that can be used to augment its functionality with a rear transition module (RTM) should that need arise.The ATCA specification provides for an RTM as a means to add used defined I/O to a board should the need arise.Aside from the programming logic's MGT lanes there are twenty-six 250 Mbps high density I/O pins plus three 1.25 Gbps high performance clock compatible differential pairs routed to the RTM connector.Together with the MGT lanes this can be used to extend and alter the functionality of the digitizer board should a different architecture of the data acquisition system be required.The availability of expansion I/O is one of the main factors contributing towards the flexibility and adaptability of the design.
There are multiple processing system peripherals present on the SoC.Of all the options available we chose to use two Ethernet controllers one SD card controller and a serial port.While none of these are required for the operation of the data acquisition system, they can be used to significantly extend its functionality.The Ethernet connections are convenient for system management, remote operation and supervision.There are two Ethernet ports on the board, one of them is on the front panel and is meant for debugging during firmware development and testing.The other one is connected to the ATCA backplane on its base channel and is meant for remote operation and management during production use.The SD card controller is connected to a micro SD card slot that can be used to extend the amount of storage available on the digitizer boards; it can also serve as an alternate SoC boot device should the internal storage get corrupted or erased.Although the write speed to the SD card is too slow to store experiment data in real time it nevertheless could be useful as a redundant, local archive of experiment data.Finally, the serial port is useful to debug the operating system running on the processor system of the SoC.

IV. CONCLUSIONS
We presented the design of a digitizer board optimized for the acquisition of magnetic signals in fusion experiments.Our design ,in particular, was tailored to the requirements of COMPASS-U.The design of the digitizer revolves around the use of dual channel isolated ADC modules that were purposely developed for digitizing signals from inductive probes while integrated in an acquisition system using digital integration.The hardware we designed allows for an acquisition system with real-time processing capabilities, with up to 576 channels in a single ATCA chassis that is galvanically isolated from the real-time control system.
We manged to harness the capabilities of modern FPGA hardware to design a flexible modular system that satisfies the requirements of COMPASS-U for channel count, bandwidth, latency and isolation.also can easily be reconfigured to satisfy the requirements of other high channel count and data volume data acquisition systems at a low cost per channel.The key behind this achievement is the use of the K26 SoM as the main processing element which allowed us to reduce the design effort as the module integrates the SoC along with power converters DRAM and storage at an accessible price point.

Fig. 1 .
Fig. 1. High level schematic of the magnetic data acquisition system.The bandwidths shown in the digitizer boards are calculated for dual channel 18bit 2MSPS ADC modules.We see the 1.8 Gbps demanded by 24 of these modules comfortably fit within the 16 Gbps available on the PCIe link.

Fig. 2 .
Fig. 2. Schematized design of the acquisition board, illustrating the connections between all major components.
The analog signals enter the board through the front panel on DD-50 connectors.The signals are routed to the modules' analog connectors using a stripline structure with dedicated shielding planes on both sides of the differential pair carrying 3 EPJ Web of Conferences 288, 03005 (2023) https://doi.org/10.1051/epjconf/202328803005ANIMMA 2023

Fig. 3 .
Fig. 3. Scheme of the clock/trigger distribution circuit.PL refers to the programmable logic fabric of the SoC.MGT refers to the multi-gigabit transceivers available on the K26 SoM, RTM is the rear transition module, used for extra user defined I/O there are 7 clock inputs from the RTM: three are connected to High Performance I/O pins and four are connected to high density I/O pins on the SoC.We use a dual PLL to clean jitter in the incoming PCIe reference clock and to reduce the jitter of the ADC conversion start signal.