Operation of the enhanced ATLAS First Level Calorimeter Trigger at the start of Run-2

In 2015 the LHC is already operating with a higher centre-of-mass energy and proton beam luminosity. To keep high trigger efficiency with the increased event rate ATLAS Level-1 Calorimeter Trigger electronics have been re-designed or newly introduced (Pre-Processors, Merging Modules and Topological Processors). Additionally, to achieve the best possible resolution for the reconstructed physics objects, complex calibration and monitoring systems are employed. Hit rates and energy spectra down to channel level, based on reconstructed events, are supervised with the calorimeter trigger hardware. In this paper the performance of the upgraded Level-1 Calorimeter Trigger at the beginning of LHC Run-2 is described.


Introduction
ATLAS is a Large Hadron Collider (LHC) experiment located at the European Centre for Nuclear Research (CERN) in Switzerland [1]. Due to the expected increase of the LHC instantaneous luminosity up to 1.6 · 10 34 cm −2 s −1 and the centre of mass energy (from 7 T eV to 13 T eV ) in the so called Run-2, starting in 2015, the ATLAS Level-1 calorimeter trigger (L1Calo trigger) needs to cope with a higher background while maintaining triggering efficiency. The L1Calo trigger designed and commissioned for LHC Run-1 conditions, made selections on missing transverse momentum (E miss T ) and multiplicity of objects (e.g. numbers of jets at various thresholds) identified in the calorimeters. Because the system as a whole has to withstand a factor of 5 higher input rates, the upgrade necessarily involved all the L1Calo components. These proceedings focus on the performance of the upgraded hardware installed during the LHC Long Shutdown LS1, during the initial phase of Run-2.

ATLAS data acquisition system
To have a better overview of what kind of role L1Calo plays it is helpful to present it in a context of the ATLAS data acquisition system (DAQ) at the beginning of Run-2. The ATLAS DAQ scheme used during the current run, is shown in figure 1. On the left side of the block diagram is presented a trigger system responsible for the event selection. It is divided into two parts. The L1 section (Level-1) filters data from the ATLAS detector using signals from the calorimeter and muon systems. Calorimeter signals arriving at L1Calo are referred to as towers, each of which is a timed sum of multiple calorimeter cells produced in several layers and combined in coarse granularity of approximately 0.1∆η × 0.1∆φ. After acceptance of the event, readout data is forwarded to a High Level Trigger (HLT) where more complex algorithms are executed to confirm the usefulness of the data. In the centre of figure 1 is shown how data flows from the ATLAS detector front-end electronics, through the readout system to storage devices.
-1 - . On the left side the trigger system is presented with the Level-1 at the top and HLT at the bottom. The components upgraded for Run-2 are outlined with green boxes, and the future ones with red. In the centre are the data flow components, from detector front-end electronics to the storage. Pre and post-LS1 high level operational parameters are listed rightmost [2].
A few factors influenced the design of the upgraded L1Calo system. Among them the most important are: decreased bunch crossing interval from 50 to 25 ns, increased luminosity which is expected to reach 1.6 · 10 34 cm −2 s −1 and the need to improve selection and quality.

Hardware improvements
The focus of this section will be on describing the improvements of new hardware relative to that already existing in Run-1 and on understanding how those result in a better event selection.

New digitisation module
The first element, which digitises data from the calorimeter, is the PreProcessor sub-system. In Run-1 it consisted of 124 VME64x modules, each hosting 16 MCMs (Multi Chip Module) [3]. The tasks of the MCM were to: sample calorimeter signals with phase adjustment (40 M SPS), synchronise pulses such they are aligned with the corresponding collision/Bunch Crossing (BC), calculate E T for the corresponding BC through the use of matched FIR filters, suppress noise and subtract pedestals for E T correction, readout the event data and send data and sum of each of four trigger input channels to the next stages of the L1Calo system.
The new Multi Chip Module (nMCM) [4] is shown in figure 2 and it consists of 80 M H z ADCs, PreProcessor FPGA and signal generator. The first noticeable advantage, comparing with the old MCM, is that the ADC samples the calorimeter analogue signal at twice the rate, which improves the handling of saturated pulses. The sampling clock delivered by the FPGA to the ADC (AD 9218) has to be shifted in phase in order to sample calorimeter signals at the proper time. Clock -2 -

JINST 11 C02059
Signal Generator 80MHz ADC AD9218 PreProcessor FPGA Figure 2. Photograph of the nMCM module mezzanine with its main components labelled [4].  phase is shifted with the help of an FPGA internal clock manager where previously an external ASIC PHOS4 was used [3]. Additionally, in contrast to the previous version of MCM, on-board signal generation for test and calibration purposes has been provided by means of custom electronic circuit (figure 2 -Signal Generator).

MET Threshold [GeV
The nMCM in comparison to the old MCM module has increased functionality also in other areas. It has better pile-up suppression and noise filtering. This was made possible by implementing programmable auto-correlation FIR filters instead of matched filters. The auto-correlation filters are necessary to deal with the increase of pileup with higher luminosity. Otherwise it would be necessary to increase thresholds which would correspond to a deterioration of L1Calo trigger quality. Additionally the nMCM has higher output data throughput -960 M bit/s (before 480 M bit/s), which is important in order to provide readout for pedestal correction information and for triggering optimised individually for Jet and e/gamma triggers. All this was possible by means of incorporating a Spartan-6 FPGA into the nMCM board and the use of its adaptive resources. In total there are 2048 nMCM boards installed and fully operational in the L1Calo system and all MCMs have been exchanged with nMCMs.
In figure 3 is shown an example of the improvement of the rate for E miss T triggers due to better pile-up suppression and noise rejection when using auto-correlation filters (pink and red lines, red is also with pedestal correction) compared with the old system (matched filters used during Run-1 -blue and black lines).

Common merger module -Extended
After energy signals are digitised, and assigned to the appropriate bunch crossing within the nMCMs, the signals are passed to Jet/Energy and Cluster Processors (JEP, CP). The CP subsystem is responsible for identifying electron/photon and tau signatures and the JEP sub-system searches for local maxima which are recognised as jets and also calculates global energy sums. This information is passed to the CMXs (Common Merger Module -eXtended) through VME backplane lines (400 per VME crate). These are also new modules incorporated into the L1Calo system which have replaced the CMMs (Common Merger Module). The replacement has been necessary primarily in order to provide information to the L1Topo sub-system. In order to provide coordinate as well as hit information to the CMX, the data rate on the VME backplane has increased from 40 M bit/s to 160 M bit/s on a single line. This provision also required firmware updates on the CP and JEP sides. The CMX boards differ only in the type of firmware which is specific to different types of data (energy, jet, electron and tau). In addition, firmware is provided for two configurations: crate CMX and system CMX. Crate CMX boards use information from one or two quadrants of the calorimeter detector and system CMX boards merge L1Calo data from the whole L1Calo system. The information transferred to and processed by the CMX boards is in the form of Trigger Objects (TOBs). A TOB specifies a Region of Interest (RoI) along with its associated transverse energy, and is likely associated with a particle produced by a collision. Based on this information the system CMX creates a total of 97 trigger signals. These signals are created when certain energy thresholds of identified objects are exceeded. This is to be compared with the Run-1 system, which featured only 56 such signals. Each CMX sorts and forwards received TOBs to the newly installed -4 - topological processor (L1Topo) via 24 optical links, each running at 6.4 GBit/s. The new CMX was commissioned with early Run-2 data. All new fast connections to L1Topo are also fully operational.

Topological processor
During the design phase of the Run-2 system it was recognised that L1Topo [6] is required to efficiently select interesting physics while maintaining L1 output rate at 100kHz with increased luminosity. The L1Topo selects events based on the geometric (∆φ, ∆η, ∆R = η 2 + φ 2 ) and kinematic relationships between TOBs. It also creates compound triggers taking into account e/γ, jets, µ, τ, E T and finally combines information from the calorimeter and muon system.
To create such a sub-system, it has been necessary to build a custom made board, or blade in ATCA terminology, consisting of 2 Xilinx Virtex 7 690T FPGAs with 80 multi-gigabit receivers per FPGA (successfully tested up to 13 Gbit/s). The sub-system is able to accept all TOBs transferred from CMXs and from the muon trigger-system interface (muon to CTP interface, MUCTPI) via the muctpi2Topo interface. This is a maximum of 216 TOBs. Additionally a Kintex 7 FPGA is used for control and data transmission. The L1Topo has an ATCA form factor as shown in figure 5. The whole sub-system comprises two L1Topo boards and is designed to deliver decisions from up to 128 algorithms. Currently the list of algorithms numbers 94 and all have been implemented and simulated both from the firmware and Monte Carlo simulation point of view. These algorithms allow to improve trigger efficiency for B , Higgs, Z and electron/photon physics cases. One has to emphasise that results of real time algorithms have to be calculated within 100 ns with a fixed latency. It is all possible due to the vast resources offered by Virtex 7 FPGA devices. In figure 5 an example of the improvement from using topological processing is shown. Here a direct cut on the φ distribution between leading jet and E miss T has been applied. Otherwise, in this spectra, dominant background from di-jet would overshadow Z H → ννbb decay. The Run-2 algorithms are currently being commissioned with physics data.

Calibration and monitoring
Improvements in trigger efficiency and better L1Calo performance wouldn't be possible without a properly working calibration chain [7]. It can be divided into different aspects or areas.
• Analogue path -calibration related to the front-end electronics: adjustment of the timing to align signals in time to remove the influence of variations in time-of-flight and the length of differential cables which range from 30 to 70m.
• Calibration related to bunch crossing identification -the digitised signal has to be properly allocated to a corresponding BC. In this case auto-correlation FIR filters are used, which helps to reduce sensitivity to pile-up effects. Inaccurate corrections would lead to poor energy estimation or hit loss in the case of small or large mistiming respectively. The process of BC identification calibration has been conducted with the aid of different tools. At the beginning of Run-2 old calibrations from Run-1 were checked. The next step of calibration was done with beam splashes -the LHC beam hit a collimator generating large energy deposits in the whole ATLAS detector. To derive the final calibration, physics signals from the calorimeter were fitted to pre-defined shapes to derive the timing offset of the signal relative to the beam as it arrives at L1Calo. The result is a timing accuracy on the level of 2ns.
As previously mentioned the BC identification is done by auto-correlation filters. One sees improvement in the BC identification efficiency in various parts of the calorimeters with these filters. This is more apparent for the end-cap and forward calorimeter where signal pile-up is more significant (figure 6 auto-correlation filters -red line, matched filters -blue line). These results have been obtained from Run-2.
• Energy calibration -pedestals parameters, auto-correlation filter coefficients and amplifier gains are derived for each tower independently in order to assure proper energy measurement. The gains are checked on a weekly basis via special pulser runs where signals with predefined but varying amplitudes are injected into the front-end calorimeter electronics. Later on comparison between L1Calo trigger tower readout and calorimeter cell readout is done -6 - offline. This analysis permits the calculation of calibration factors and based on this proper gain values are set in the hardware. As a result it is ensured that ADC values are proportional to E miss T -1FADC ≈ 0.25GeV . This process of calibration also permits to localise dead/noisy channels.
In figure 7 we show a plot of normalised trigger rate versus average instantaneous luminosity per bunch. It is evident that the expected linear correspondence between these two values is nearly followed only in the case of pedestal correction enabled (blue line) and not in the case of pedestalcorrection disabled (red line). This shows the importance of the pedestal correction in the nMCMs and its influence on L1Calo trigger quality.
L1Calo monitoring is crucial not only in conducting thorough checks of the system during its operation but also essential in the integration of new hardware into the L1Calo ATLAS trigger system. Monitoring is divided into several parts. System monitoring is connected directly to the L1Calo hardware and reads out its parameters. Then on-line monitoring samples data streams coming from the L1Calo system and finally offline monitoring analyses all recorded events. Monitoring at these different stages delivers important information about the state of the L1Calo system in the form of hit-maps, energy distributions and comparison to bitwise-simulation of the real-time data path. This represents detailed information about hardware, firmware status, transmission errors and online-software performance. Monitoring, therefore, also provides the means to simulate real-time L1Calo data path and compare it to collected data.
The monitoring had to be also adapted to the new conditions in Run-2: new hardware and functionalities implemented in the L1Calo system. The most important improvements are related to: channel-wise and trigger independent rates and spectra, Pre-Processor Module rate-metering and histogramming and of course the new hardware itself (CMX, L1Topo). In figure 8 an example of online monitoring, where it is possible to compare data consistency of the data sent from CP to the CMX sub-system, is presented.

Conclusions
Improvements in trigger efficiencies and better performance of L1Calo system wouldn't be possible without LS1 upgrade. It provides now new functionalities and possibilities: • 2048 MCMs have been replaced by nMCMs. This not only provides the previous functionality but also improves timing, energy and reduces the influence of pile-up/noise on the trigger, • JEP and CP firmware have been modified to transmit not hit-counts but TOBs along their custom backplane, • new merging modules (CMX) receive the transmissions from the CP and JEP modules, provide the thresholding for an extended number of thresholds (in comparison to Run-1) and forward the TOBs to the L1Topo sub-system, • the L1Topo is a new sub-system for real-time event selection based on geometrical and kinematic quantities. The L1Topo sub-system is nearing the end of commissioning, • the calibration and monitoring tools, as adapted to Run-2, provide means needed for proper integration of the sub-systems into the ATLAS trigger system, and play a significant role in maintaining stable operation of the L1Calo system as a whole.
As LHC luminosity continues to climb, even better strategies requiring more radical changes to the L1Calo system will be required. In that vein, additional programmes aimed at improving the trigger performance and effectiveness of its selection are already underway [8].