The NA62 Liquid Krypton calorimeter readout architecture

The NA62 experiment [1] at the CERN SPS (Super Proton Synchrotron) accelerator studies the ultra-rare decays of charged kaons. The high-resolution Liquid Krypton (LKr) electromagnetic calorimeter of the former NA48 experiment [2] is a key component of the experiment photon-veto system. The new LKr readout system comprises 14,000 14-bit ADC acquisition channels, 432× 1 Gbit Ethernet data request and readout links routed by 28× 10 Gbit network switches to the experiment computer farm, and timing, trigger and control (TTC) distribution system. This paper presents the architecture of the LKr readout and TTC systems, the overall performance and the first successfully collected experiment physics data.


Introduction
The NA62 experiment's main aim is to explore distances as small as 10 −21 m, studying ultra-rare kaon decays, in particular K + → π + νν. These rare decays are excellent processes to probe the Standard Model (SM) and search for new physics complementary to the Large Hadron Collider (LHC) studies. The experiment is housed in the CERN North Area on a new dedicated high intensity beam line, where 400 GeV/c protons, extracted from the SPS accelerator, produce a secondary charged hadron beam by impinging on a beryllium target. In view of the low branching fraction of the decay process under study imposes a background rejection factor at a level of 10 12 , a precise measurement of the event kinematics, hermetic photon vetos and particle identification capabilities are key requirements of the experiment. About 100 K + → π + νν decays are expected to be collected by the NA62 during 2-year run with nominal intensity.

NA62 detectors layout
The experimental subsystems are spread along a 170 m long region starting about 100 m downstream of the beryllium target. After the beryllium target, the experiment receives a high intensity hadron beam with fixed momentum (75 GeV/c) and an average content of 6% kaons. This secondary kaon beam line leads to the fiducial decay region located in a ∼ 117 m long and a 2.4 m in diameter (on average) vacuum pipe.
The NA62 configuration is inspired by the experience gained with the previous kaon decay experiments performed at CERN. The experiment is consists of tracking devices for K + and π + identification and calorimeters in order to veto photons, positrons and muons. A schematic layout of the experiment is shown in figure 1, more details can be found in [1].

The experiment data acquisition (DAQ) system
The NA62 experiment consists of 12 sub-detector systems and several trigger and control systems, for a total channel count of about 100,000. The average event rate integrated over the experiment's detectors is about 10 MHz and only a very small fraction of these events contains data relevant to the physics goals. Therefore a multilevel trigger structure is being implemented in order to reduce this rate to a few kHz. The experiment's trigger and data acquisition system is presented in figure 2.
The first level trigger (hereafter called L0) selects events after the processing of trigger primitives prepared with the data from participating detectors. The latency of this processing has an upper limit of 10 ms. The L0 signal is distributed to the sub-detector readout electronics via the timing, trigger and control (TTC) links (see section 1.3).
The next trigger level (called L1) is software based and used by some sub-detector systems which are not read out at L0. Subsequent L1 requests are put in Ethernet packets in an asynchronous way within the trigger data processing latency of about 1 s. Upon the reception of a L1 request, each contributing sub-detector sends its data to a farm of PCs for further decisions.
The last trigger level (L2) is based on correlations between different sub-detectors' L1 data. The information, upon which these correlations are determined, is provided by an event-building PC farm. The latency of the L2 trigger is not fixed and can extend into the SPS inter-spill period.

Timing Trigger and Control (TTC) system
The experiment data taking sequence is defined by the SPS accelerator cycle and entirely driven by the TTC system [3] that was developed for the LHC experiments and is used by all NA62 sub-detectors.
This TTC system is a unidirectional optical fibre based transmission system, where two information channels, A and B, are Time Division Multiplexed (TDM) and Bi-Phase Mark (BPM) encoded using the LHC Bunch Crossing (BC) 40.08 MHz clock (hereinafter called the 40 MHz clock) as the carrier frequency and transmitted with a rate of 160 MHz. One channel (A) carries -2 - exclusively the trigger accept signal (L0) and the other (B) carries packaged address and data information for the sending of various commands and parameters.
The sub-detector-specific interfaces receive all timing signals via the experiment optical distribution network and convert, decode and deliver these signals to all sub-detector data acquisition electronics.

LKr calorimeter and its readout electronics 2.1 Calorimeter
The LKr calorimeter is a quasi-homogeneous electromagnetic calorimeter which ensures a very good intrinsic energy resolution. It is a key element for vetoing photons from K decays, with the requirement to have a photon detection inefficiency lower than 10 −5 for energies larger than 35 GeV. In addition, the calorimeter is required to provide trigger signals based on energy deposition for reducing the L0 trigger rate.
The calorimeter active medium consists of a bath of ∼ 10 m 3 of liquid krypton at 120 K with a total thickness of 125 cm (∼ 27 radiation lengths) and an octagonally shaped active cross-section of 5.5 m 2 . An 8 cm radius vacuum tube goes through the centre of the calorimeter to transport the undecayed beam. Thin copper-beryllium ribbons (of dimensions 40 µm × 18 mm × 127 cm) stretched between the front and the back of the calorimeter form a tower-structure readout. Each -3 -of the 13,248 readout cells has a cross-section of about 2 × 2 cm 2 and consist (along the horizontal direction) of a central anode (kept at high voltage) in the middle of two cathodes (kept at the ground). More details about the LKr calorimeter structure can be found in [4].

Front-end electronics
The front-end part of the calorimeter readout was built for the NA48 experiment and comprises two circuits. The initial signal is derived from the charge measured by a preamplifier mounted inside the cryostat at liquid Kr temperature and connected to the anode electrode by a coupling capacitor. The integration time constant of the charge preamplifier is 150 ns. The signal from the preamplifier is transmitted to a combined receiver and differential line driver mounted outside the calorimeter close to the signal feed-through connectors. The receiver amplifies the preamplifier signal and performs a pole-zero cancellation. The signal after pole-zero cancellation has a rise-time of about 20 ns and a fall-time of 2.7 µs. The maximum signal level, 50 GeV, corresponds to ±1 V into 100 Ω at the digitizer electronics input. The obtained signal to noise ratio is 15,000 to 1.

Back-end electronics
The initial back-end parts of the readout chain, TIC and CPD [4], were developed in 1995. At the design time, only a few low cost 10-bit 40 MS/s FADC were available and a custom dynamic range switching ASIC was developed to fulfil the experiment requirements. Additionally, the maximum event readout rate was 13 kHz, almost two order of magnitude less than for NA62. Therefore, two front-end elements which fulfil all requirements have remained untouched for the new experiment, but the initial back-end performance was not compliant and had to be upgraded. New LKr readout system has a modular multi-channel architecture based on 14-bit ADC, high-capacity DDR3 memory modules and high-speed serial links. The data requests and readout links are routed by Gbit network switches to the experiment computer farm. In addition to the data processing and readout, the LKr digitised signals from the selected channels are summed to build super-cells and delivered to the experiment trigger system.

JINST 11 C01070
The rate of kaon decays reaching the detector is about 10 MHz and only a very small fraction of these events is targeted. Therefore, the data taking sequence is driven by high-performance triggering system based on the TTC system. The LKr-specific TTC-LKr interface receives all timing signals via the optical distribution network. It then converts, decodes and delivers these signals to the calorimeter data acquisition electronics.

Calorimeter REAdout Module (CREAM)
The new back-end acquisition board architecture is based on modern technologies. While all documents and comprehensive specification were prepared at CERN, due to the huge number of modules (∼ 14k channels) needed to instrument the LKr readout and the maintenance requirement over the lifetime of the experiment (∼ 10 years), the development and production was sub-contracted to a commercial vendor [5].
The CREAM is a 1-slot wide VME 6U form-factor module. The signal is shaped before the ADC input into a differential semi-Gaussian signal with a 40 ns rise time and a 70 ns full width at half maximum (FWHM). A 14-bit DAC (AD5648) allows tuning the DC offset of each channel in order to correctly adjust the pedestals and to preserve the dynamic range. One module houses 4 8-channel, 14-bit, 50 MSPS ADCs (AD9257 [6]) with an on-chip sample-and-hold circuit and one serial output data link per channel. Table 1 gives the main parameters of the analog to digital conversion of the module including the input shaper. The input data circular buffers, as well as the L0 event buffers, are implemented in one 8 GB storage capacity DDR3 SODIMM module (HYNIX HMT41GS6MFR8C-H9). The CREAM functionalities are mainly implemented in an Altera Stratix-IV FPGA (EP4SGX180KF40C4). The module can run with external and internal clock sources, and the experiment default sampling frequency is 40.08 MHz. The external reference sampling clock is provided by the TTC links. The data processing flow with multiple levels of triggering can be summarised as follows: • analog inputs, after proper shaping, are continuously digitised using the 40 MHz clock; • trigger sums are continuously formed in digital form and sent to the L0 trigger logic; • data are continuously written in a circular buffer waiting for the L0 decision; -5 -• upon receipt of a L0, the related data, stored at fixed latency time before L0, are extracted from the circular buffer and stored into another buffer called L0 event buffer, waiting for a possible L1 trigger request; • upon receipt of a L1, the corresponding data are sent to a PC farm through a gigabit Ethernet port (see section 2.4 for more details about the data requests and readout).
Since for each event a large fraction of channels only contains pedestal counts, a zero suppression algorithm of individual channels is implemented. The flexibility of the CREAM architecture can be exploited also for an alternative, triggerless mode of readout. In this scheme, continuously digitized data are analysed by a pipeline process running on the FPGA. Precise information on the energy and time of each pulse is obtained and continuously sent to the PC farm to build complete events.
During the data acquisition, digitised signals from the selected channels are summed up to build a Trigger Sum (Super-Cell) to be sent to the LKr L0 trigger system. Up to 4 high-speed differential links are able to accommodate 4 Super-Cells per CREAM module with effective data rate per link of 720 Mbps.
The module conforms to the IEEE-1014-1987 and ANSI/VITA 1-1994 [7]. The board hosts the VME P0, P1, and P2 connectors and fits into both VME and VME64 standards, more details about the CREAM can be found in [8].

LKr Timing Trigger and Control (TTC-LKr) interface
The LHC TTC system was developed in the 90's and most of the components were designed with a technology now obsolete. Therefore, alternative solutions were considered in order to substitute the out-dated components and a common CERN-Boston University development [9] in FMC formfactor [10] was chosen for this project. The design is based on commercial off-the-shelf components and the ADN2814 receiver [11] is used as the clock-data recovery circuit (CDR), while the data decoding is done by the carrier board FPGA.
The main function of the TTC-LKr [12] is to provide 40 MHz clock, L0 trigger information, broadcasts and individually-addresses control signals for CREAM digitisers. The source of these signals can be selected from either the optical 160 Mbps BPM encoded bit-stream input, electrical front-panel inputs or an internal signal generator. Once selected, the relevant TTC source is converted, decoded and made available on the user-defined VME backplane P0 connector. All TTC signals on the P0 connector are synchronised with the 40 MHz clock which is delivered via the P0 as well.
A VME64x (IEC 1076-4-101) conformed 95-line P0 (J0) passive plug-in backplane was developed at CERN in order to distribute TTC signals via the P0 connectors. The backplane is a mixture of parallel and point-to-point topology (multi-drop differential or single-ended) and it provides a physical and electrical interconnect between TTC-LKr and CREAM modules in one crate. In order to minimise the total capacitance and optimise the signal integrity, the TTC-LKr is positioned in the middle of the crate in slot No. 11. Thus, one TTC-LKr can serve up to 19 CREAMs in the same crate.
The TTC-LKr is single-width 6U VME64x compliant unit. The TTC-FMC [13] mezzanine board is used to convert and recover the optical inputs, whereas the TTC stream decoding, further -6 - interfacing and assembly control are implemented in a Xilinx Spartan-6 FPGA. The layout of the LKr TTC distribution and the modules picture are presented in figure 4.

LKr data request and acquisition
Data requests and readout to/from the 16 CREAM modules housed in one crate (see figure 4) are provided via one 24-port HP 2920-24G (J9726A) network switch by the experiment computer farm. Thus, for entire LKr readout, 432 × 1 Gbit Ethernet CREAM ports are routed into 28 × 10 GbE optical links.
The LKr default data readout is initiated by the L1 trigger request (see section 1.2) which is sent to CREAMs by the computer farm as UDP packet. Each packet, called a Multi-Request Packet (MRP) can contain up to 100 L1 trigger requests. The routers distribute the MRPs in multicast mode to all CREAMs which have previously joined a proper multicast group by issuing the Internet Group Management Protocol (IGMP) packet. CREAM data corresponding to the requested event number, formatted as UDP packets and called Sub-Detector Events (SDE), are sent out through the same link to the PC farm. The IP address of the requesting computer is automatically retrieved from the MRP by the CREAM firmware.
In the experiment control room a farm of multi-core PCs with 10 GbE cards handles a large switch (O(128) ports) which provides all LKr links, and a matching number of 10 GbE links for the transfer of complete events to the final NA62 event building farm. This implementation is dimensioned to allow a full-rate non-zero suppressed data transfer. The baseline mode of operation is however to apply zero suppression algorithms to the events, except for a fraction of those (random events, calibrations, downscaled control triggers). The layout of the LKr network and computer farms is illustrated in figure 5.

Project status
All new LKr back-end components were installed in the experiment mid-2014. This includes common infrastructure, 28 6U crates with VME bridges and network switches, the TTC distribution optical components and TTC-LKr interfaces, and, most crucially, the complete production of CREAM modules, 450 pcs. Then, the entire LKr readout system was commissioned and first physics data of the new NA62 experiment were successfully collected in the fall of 2014 after CERN's Long Shutdown. While the 2014 run data were taken with just a few percent of nominal intensity, the achieved overall performance of new LKr readout was well beyond the experiment requirements.
Though more analyses of the NA62 data are on going, figure 6 shows the LKr cells illumination and the squared missing mass distribution for selected kaon decays reconstructed by exploiting the calorimeter data. Figure 6. LKr illumination (left) and K + → π + π 0 reconstruction (right).