Upgrades to the ATLAS Level-1 Calorimeter Trigger

In 2015 the Large Hadron Collider will run with increased center-of-mass energy and luminosity. To maintain a high efficiency in selecting interesting collisions for the physics analyses in the next data-taking period, event topology information will be added to the ATLAS Level-1 real time data path and processed by a new Topology Processor (L1Topo). To cope with the luminosity levels foreseen after the 2018 LHC upgrade, a new digital trigger path for the Liquid Argon calorimeters will provide finer granularity and depth segmentation in the electromagnetic layer to new Level-1 feature extractors (FEX) for an improved electron, photon, tau and jet selection. We present the ongoing and future calorimeter trigger upgrades to the ATLAS Level-1 trigger.


Introduction
ATLAS is one of the LHC experiments which is described in detail in ref. [1]. Luminosity upgrades to the LHC mean that the ATLAS Level-1 trigger system must deal with pileup conditions beyond the original design requirements while maintaining manageable trigger rates and a high efficiency for selecting the data of interest to the physics community. In this paper we describe current upgrade work and planned extensions for the Level-1 Calorimeter Trigger (L1Calo) that will add topology algorithm capabilities and improved trigger object identification with finer-granularity calorimeter data.

Overview of Level-1 Trigger (L1)
The ATLAS Level-1 Trigger system (L1) [2,3] performs initial event selection using reducedgranularity information from the calorimeters and muon spectrometer. Trigger result bits from the different L1 subsystems are received and combined in the Central Trigger Processor (CTP) to produce a Level-1 Accept (L1A) to initiate readout of selected events.

Level-1 Calorimeter Trigger (L1Calo)
The L1Calo system comprises three main sub-systems ( figure 1 and 2). The first of these is the PreProcessor, which receives and processes analogue tower sums from the electromagnetic (EM) and hadronic calorimeter layers [4]. The signals are first digitised and then digitally processed to provide calibrated transverse energy (E T ) sums time-aligned to the correct bunch crossing. These sums are then transmitted electrically to the Cluster Processor (CP) and the Jet-Energy Processor (JEP) subsystems [5].
The CP subsystem performs sliding-window algorithms to identify isolated electron, photon and tau candidates, while the JEP identifies jet candidates and calculates the global sums of total and missing transverse energy. For both subsystems, real-time results from the cluster-processor (CPM) and jet-energy sum modules (JEM) are gathered over a high-density backplane to the socalled Common Merger Modules (CMMs), which report threshold bit multiplicities and energysum thresholds to the Central Trigger Processor. Upon an L1A, the JEM and CPM modules read out the coordinates and thresholds passed by each identified candidate to the high-level trigger as Regions of Interest (RoIs).

Level-1 Muon Trigger (L1Muon)
The Muon trigger system receives hit information from Resistive Plate Chambers (RPC) in the barrel region and Thin Gap Chambers (TGC) in the forward regions. Track segment formation and final muon candidate selection depending on the transverse momentum threshold are performed by on-detector sector logic. Sector logic results are sent to the CTP via the Muon CTP Interface (MUCTPI) [6], which performs the final combinatorial logic, and which also reads out the muon RoIs and passed them to the high-level trigger.

Central Trigger Processor (CTP)
The Central Trigger Processor (CTP) [7] produces the final Level-1 Accept L1A for reading out selected events using combinatorial logic applied to the Calorimeter, Muon and ancillary trigger inputs as controlled by the trigger menu.

L1Calo Trigger upgrades
Several upgrades to the current L1Calo system are already underway during the 2013-14 long shutdown (LS1). The first, not covered in this paper, is an FPGA-based replacement for the frontend multichip module (MCM) [8] for the L1Calo PreProcessor that will provide improved noise performance and more sophisticated digital filtering at higher pileup conditions.
Upgrades to the digital algorithm processors will add topology information to the trigger realtime data path. FPGA Firmware upgrades on the CPM and JEM algorithm processor modules will increase the real-time output rate to the backplane by a factor of four, so that instead of threshold multiplicities, the modules transmit more detailed "trigger object" (TOB) information. The original merging modules (CMMs) will be replaced by new, extended functionality (CMX) modules capable of receiving the higher-speed backplane data and performing local multiplicity processing with more thresholds, as well as sending the TOBs over high-speed optical links to a new topological processor subsystem (L1Topo) subsystem.
L1Topo will receive TOBs from the CP, JEP and L1Muon subsystems, and is capable of performing geometrical cuts and angular correlations between TOBs, as well as calculating complex observables such as invariant mass.
-3 -For the second long shutdown in 2018 (LS2), new on-detector electronics will be installed for the Liquid Argon Calorimeter to provide a new trigger path of all calorimeter samples, allowing higher-granularity calorimeter information to be sent to L1Calo. The new off-detector digital processing chain will consist of a so-called Digital Processor System (DPS) that will perform digital filtering on the new digital calorimeter data, and two new L1Calo "feature extractor" subsystems, one for improved electron/hadron identification (eFEX) and one for jet identification (jFEX).

Topology upgrades to L1Calo
The original L1Calo architecture includes only multiplicities of identified jets and clusters in the real-time data path, while their coordinates and thresholds are stored locally in pipeline buffers and read out to the Level-2 trigger as RoIs after a Level-1 accept. "Common Merger Modules" (CMMs) in each crate gather real-time results over parallel point-to-point backplane links at 40 Mbit/s to produce crate-wide results. The CMMs in one crate of each subsystem also function as system-wide mergers by gathering crate-level results from other CMMs over parallel LVDS cables to produce global results that are sent directly to the CTP.

CPM and JEM firmware upgrades
Studies of the custom processor backplane in the Cluster Processor (CP) and Jet/Energy Processor (JEP) subsystems have shown that data transmission at much higher rates than the nominal 40 Mbit/s is possible without loss of data integrity. The legacy FPGAs on the CPM and JEM modules place a limit of about 160 Mbit/s. By expanding the real-time output bandwidth from 40 to 160 Mbit/s it is possible to transmit detailed information for up to four jet "trigger objects" (TOBs) or five EM/tau TOBs onto the backplane for each bunch crossing. EM and tau TOBs nominally includes an 8-bit E T sum, up to five isolation thresholds and eta-phi coordinates. Jet TOBs include coordinates and 9-10 bit E T sums for two different jet cluster sizes.
Currently, a new jet identification firmware in the JEM is ready for initial tests, while the energy summation and the CPM algorithm upgrade is in progress. No hardware modifications to the JEM or CPM are needed, and there are sufficient FPGA resources to implement the new features.

Extended common merger module (CMX)
As mentioned above, the CMM will be replaced by a new module (CMX) that receives the higherspeed backplane input data, produces legacy crate-and system-level threshold multiplicity triggers and transmits received TOBs over high-speed optical links to the Level-1 topology processor (L1Topo) [9]. An optional, simplified topology-processing capability is included on some CMX modules to serve as a possible supplement to or as a temporary replacement for L1Topo.
The CMX is a 9U module that is mechanically and electrically compatible with the existing CMM. All real-time processing (crate, system and topological) is performed in two separate large FPGAs (Virtex-6). Readout to the DAQ and RoIs is through optical links driven by emulated Glink serialisers in the FPGAs implemented using GTX transmitters clocked at 960 Mbits/s. The CMX includes several parallel-optical modules, driven by dedicated multi-Gbit transceivers on the -4 - FPGA, nominally designed to run at 6.4 Gbits/s. Up to two 12-fiber ribbon transmitter modules can send TOB data to multiple L1Topo modules as needed, while up to three 12-fiber ribbons receiver modules can receive incoming TOB data for topology processing.
The CMX prototype is expected to be available for initial tests at the end of December 2013. Extensive hardware and firmware tests are scheduled to be done at Michigan State University (MSU) and CERN. The final CMX fabrication and integration with the existing system will be completed in the second half of 2014.

Level-1 Topology Processor (L1Topo)
L1Topo is a single-crate system, nominally equipped with two identical ATCA-compliant modules ( figure 3). A third module can be added in a later stage depending on the number and complexity of topological algorithms required. L1Topo receives TOB data (Trigger Object data) for jets, clusters and muons over parallel-optical ribbon fibres. and processes them in two large Virtex-7 FPGAs per module, each with up to 80 Multi-Gigabit Transceivers (MGT). L1Topo results are sent to the CTP via electrical and/or optical links.

Real-time data path
Input data are received optically through four blind-mate MTP/MPO connectors in ATCA Backplane zone 3 of L1Topo (the backplane definition is divided into three sections: zone-1, zone-2 and zone-3). In the baseline design 48-way connectors will be used. The optical signals are distributed -5 -over octopus cables to 12-fibre optical-electrical (o/e) receivers on the main module. For reasons of design density miniPOD receivers are used. The o/e receivers are rated up to 14 Gb/s, and placed as close as possible to the FPGAs to optimize signal quality.
The Multi-Gb/s Transceivers (MGT) on the FPGAs will be run at an initial line rate of 6.4 Gb/s per channel, although higher rates are possible. High bandwidth, low latency parallel data paths between the two FPGAs using GPIO are provided for real-time communication between the two processors, but before LS2 (ATLAS upgrade, 2018) each FPGA will be capable of receiving the full event topology information optically, and the two processor FPGAs will process data independently and in parallel.
Output to the CTP will typically consist of individual result bits indicating whether a specific topology algorithm passed or not. The resulting trigger data will be transmitted to the CTP optically by optical fibres or parallel electrical cables via an extension mezzanine module.

Implementation details
L1Topo is designed to work with a choice of footprint compatible devices. The prototype module has been outfitted with two less-powerful XC7VX485T devices due to component availability. More powerful devices (XC7VX690T) with the full 80 input MGTs will be mounted on the production modules. The backplane optical connectors provide a capacity of up to 288 fibers, and four 48 way fiber bundles are foreseen.
For synchronous operation, data transmitters will have to be operated with clean multiples of the LHC bunch clock. Receiver reference clocks are segmented and derived from local crystal oscillators. The L1Topo module is designed for 40.0789 MHz operation of the real-time data path only. The fabric clocks run at the LHC bunch clock frequency. The jitter on the MGT bunch clock path is tightly controlled with help of a PLL device. The control FPGA receives additional local clocks since it handles DAQ, ROI, and control links as well.
A single fibre optical ribbon connection per processor FPGA, running through the front panel of the module is provided for optical result transmission to CTP.
Currently, the L1Topo prototype test results are promising. The final production and integration with the existing system are also scheduled for the second half of 2014.

ATCA pre-configuration and control
An initial step of module initialization is performed by an IPMC device. It communicates to the shelf via an I2C port (IPMB) available on all ATCA modules in zone 1 (see below). There is a IPMC based controller available on L1Topo. A local SPI memory will allow for an alternative configuration method for the control FPGA. The processor FPGAs will be configured off an SD flash memory, which in turn is sequenced by the control FPGA. In-situ configuration update will be possible via IP access to the control FPGA. IP connectivity is provided by two 10/100/1000 Ethernet ports on the backplane in zone 2 (redundant base interface), and IPbus (an IP-based control protocol for ATCA) is used for nominal module control and monitoring once the control FPGA is configured.

LS2 upgrades
After shutdown in 2018, the accelerator luminosity will increase from 2 to 3 × 10 34 cm −2 s −1 and associated pileup will also increase. To manage the increasing rates in this regime, whilst preserving an effective and efficient calorimeter trigger, significant upgrades will be necessary. Specifically, more refined processing of electromagnetic calorimeter information, at higher granularity, will enable better discrimination between photons, electrons, taus and jets and hence provide efficient triggers for electroweak-scale physics. Improved processing, also at higher granularity, will enable better reconstruction of jets and, using a wider search window, will open up the possibility of identifying broader jets.
An overview of the Level-1 trigger system for LS2 is shown in figure 4, with L1Calo set in the context of other system components. For LS2 Liquid Argon Calorimeter (LAr) signals will be digitized at detector frontend and sent to a new Digital Processing System (DPS) to produce E T measurements at a finer granularity. Information from the Tile Calorimeter will be derived from the existing digitisation system (within L1Calo) at the maximum granularity available before LS3 (ATLAS upgrade, 2022-2023). These data will be sent to the new electromagnetic and jet feature extractor subsystems (eFEX and jFEX respectively) over multi-gigabit optical fibre links. The eFEX subsystem will employ new cluster-finding algorithms on the higher granularity data to produce more refined TOBs. The jFEX subsystem will use jet-finding algorithms on higher granularity data. Table 1 contains the principal system parameters for eFEX and jFEX subsystems. Both eFEX and jFEX systems receive digitised E T values on optical fibres. The current baseline assumes 6.4 Gb/s link speed. In order to reduce module complexity and extend the system capabilities, the higher use of link speeds (∼ 10 Gb/s) is under consideration and will be adopted if tests prove successful.

eFEX/jFEX functionality
The eFEX and jFEX modules have similar architectures, differering mainly by the number and connectivity of optical input links and algorithm processor FPGAs. Each module has a "core" angular coverage for trigger object selection algorithms, and needed environment data from neighboring regions are provided as duplicated optical links, realized through a combination of active electrical duplication and passive optical splitting.

Real Time Data path
Similarly to L1Topo, input data enter the FEX modules optically through the backplane zone 3 via four MTP-style optical backplane connectors with up to 72 fibres each. "Octopus" cables divide the input fibres into groups of 12 and bring them to o/e converters, which are currently foreseen to be Avago MicroPOD devices.
After just a few centimeters of electrical trace length, the multi-gigabit signals are de-serialised in the processor FPGAs. The exact data formats are still undefined, though standard 8b/10b en--8 - coding is envisaged for purpose of run length limitation and DC balance. Data sharing between neighbouring FPGAs is needed, and may be implemented by some combination of passive splitting, high-speed fanout buffers, and/or the use of "PMA loopback" in the FPGA MGTs. There is a latency penalty of about one LHC bunch clock associated to the latter.
Resulting trigger objects are sent on parallel fabric I/O to a merger FPGA, which reports results to the L1Topo processor.

High Speed link simulation and test
The eFEX and jFEX designs are challenging to design and build, and reliable board-level simulation of the high-speed signal traces is essential.
A high-speed demonstrator was developed at the Rutherford Appleton Laboratory with a new systematic design methodology based on high-speed PCB simulation, which has been successfully validated at 10 Gb/s with oscilloscope measurements. The left picture of figure 5 shows the eye diagram measurement at 10 Gb/s on oscilloscope, the right picture shows the eye diagram simulation overlaid on measurement. There is an excellent correlation between PCB channel simulation and oscilloscope measurement. Figure 6 shows the rate (BER) scan test results with several Xilinx Virtex-7 GTH/MicroPOD channels. It has achieved over 40% margin at a BER of 10 −12 consistently. This is a very encouraging result.

Conclusions
In these proceedings the upgrades to the ATLAS Level-1 Calorimeter Trigger are described. Over the 2013-14 shutdown (LS1), firmware and hardware upgrades to the existing Level-1 Calorimeter Trigger (L1Calo) will add object coordinates and ET to the real time data path, and a new topology processor (L1Topo) will be added to Level-1. The upgrade to L1Calo includes upgraded CPM and JEM firmware to send trigger object data across the backplane at increased data rates. The current CMMs will be replaced by new CMX modules capable of receiving and processing the high-speed -9 - backplane data, and sending them over high-speed optical links to a new topological processor (L1Topo) subsystem. Real-time L1Topo output will be sent to the Central Trigger Processor CTP, where the final Level-1 accept decision is taken. The crucial aspect here is the topology algorithm development, including optimal bandwidth, FPGA resources and latency.
The luminosity of the LHC is expected to increase by a factor of 2-3 after LS2, currently foreseen for 2018. To improve the ATLAS Level-1 Trigger performance changes are needed at both the ATLAS detector and the Level-1 Trigger system. New on-detector electronics, the digital Tower Builder Board, will be installed on the Liquid Argon Calorimeter to provide higher granularity calo trigger information. New algorithms running in additional hardware will be added to the existing Level-1 Trigger to process these data. The new off-detector digital processing chain will consist of three subsystems: a Digital Processor System (DPS) will perform digital filtering on the new digital calorimeter data and a feature extractor subsystem with EM and Jet processors (eFEX, jFEX) will identify calorimeter trigger signatures. Information describing objects identified by the FEX subsystems will be sent over optical fibres to a Topology Processor subsystem. We also presented here the motivation for architectural design choices, latest test results, and high-speed link simulations.