Upgrade of the ATLAS Central Trigger for LHC Run-2

The increased energy and luminosity of the LHC in the run-2 data taking period requires a more selective trigger menu in order to satisfy the physics goals of ATLAS. Therefore the electronics of the central trigger system is upgraded to allow for a larger variety and more sophisticated trigger criteria. In addition, the software controlling the central trigger processor (CTP) has been redesigned to allow the CTP to accommodate three freely configurable and separately operating sets of sub detectors, each independently using the almost full functionality of the trigger hardware. This new approach and its operational advantages are discussed as well as the hardware upgrades.


Introduction
The ATLAS experiment [1], which is located at the Large Hadron Collider (LHC) [2] at CERN, uses a two staged trigger system to identify collision events of interest. The first stage, called Level-1 trigger [3], reduces the event rate from 40 MHz to 100 kHz using information from dedicated muon trigger detectors and from the calorimeters. It is a synchronous, pipelined system that operates at the LHC Bunch Crossing (BC) frequency of 40.08 MHz and is implemented in custom built hardware. The second stage of the trigger system reduces the event rate further to 1 kHz. It is software based and uses offline-like reconstruction algorithms, utilizing in a the information of all sub-detectors in regions of interests around the level-1 objects in full granularity. In addition algorithms using the full event information are used. In the run-2 data taking period of the LHC, starting in spring 2015, the instantaneous luminosity will be increased by at least a factor of two with respect to the LHC run-1, reaching up to 2 · 10 34 cm −2 s −1 . Furthermore, the collision energy will be increased from 8 TeV to 13 TeV yielding an increase in the hard interaction cross section of about a factor of two for many physics processes of interest. As the event storage rate is only doubled, the trigger must become more selective while sustaining the sensitivity for the physics objects of interest. This made major upgrades of the trigger system necessary to allow for more sophisticated trigger criteria. The upgrades were installed during the two year long shutdown in 2013/2014.
The first stage trigger system is depicted in figure 1 as it will be used during the LHC run-2 data taking period. All upgraded systems are indicated by dashed lines. The central trigger system receives preprocessed information from the calorimeters and muon detectors encoding multiplicities and energy/momentum information and dedicated information about (missing) transverse energy and objects identified as τ-leptons. Additional information from forward detectors, beam pickups, and minimum bias scintillators is processed as well. A fixed number of bits encoding the information are transmitted to the Central Trigger Processor (CTP) and used as inputs for the trigger decision. A new component of the system is the topological processor [4], which allows for the evaluation of topological 1 selection criteria using calorimeter and muon information. This requires 1 Topological selection criteria could be e.g. relative angular information between trigger object, derived information like the combined invariant mass of multiple trigger objects and so on.   Figure 1. Schematic overview of the ATLAS first level trigger system. The L1Muon processes information from dedicated trigger chambers of the muon system and sends pT threshold and multiplicity information to the central trigger. The L1Calo processes coarse granular information from the calorimeters and provides clusters above a given energy threshold, jet multiplicities for various energy thresholds, transverse energy sum and missing transverse energy as well as dedicated tau triggers. Transmitted to the CTP are a number of bits (18 from L1Muon, 196 from L1Calo) which encode the information. More detailed information, including η and φ coordinates are transmitted to the topological processor. The CTP combines all input signals using configurable logic rules and produces the L1 trigger decision.
additional inputs on the CTP which decides if an event is accepted at the first stage or not. In addition to the generation of the trigger signals, including the dead-time generation, the CTP is also responsible for the timing and synchronization signals which are distributed to all sub-detectors. The CTP has undergone major hardware upgrades. Furthermore, the software system controlling the CTP has been completely redesigned supporting new hardware features and in particular the partitioning of the CTP into three logically separated, independently running partitions supplying three sets of sub-detectors with trigger and timing signals. This paper focuses on the upgrade of the CTP, which is detailed in the next section, followed by the introduction of the new software infrastructure in section 3. The current status of the upgrades is summed up in section 4.

The Central Trigger Processor
The trigger path depicted in figure 2 is implemented in an FPGA located on the CTPCORE+ module, which is one of several custom made electronics boards the CTP is composed of. The digital input signals (cf. figure 3) are logically combined into 512 trigger items using look-up-tables, which can perform OR operations and the decoding of multiplicities, and a content addressable memory, which is used to perform AND and NAND operations. The trigger items are then put in coincidence with up to 16 bunch groups. Each bunch group contains a list of LHC bunches that should be taken into account. Typical lists contain all colliding, filled or empty bunches. For each trigger item the bunch groups which should be used can be configured freely. After that each item is pre-scaled by an adjustable factor, using a random pre-scaling algorithm. Each item can be vetoed by the OR of the busy signals from sub-detectors and dead-time constraints. Dead-time constraints are implemented by up to 4 leaky bucket algorithms which model the derandomizers of the detector front end electronics and a fixed dead-time after each issued trigger. This concept was used very successful during run 1. Each item can also be disabled individually via the configuration. Eventually the OR of all trigger items, called L1 Accept (L1A), is sent to all sub-detectors to trigger their readout. The block generating the final L1A signal is replicated three times in the upgraded CTP-CORE+ module, including the VETO block and dead-time generation as well as is the generation of further timing signals. This allows the operation of three independent sets of sub-detectors, called partitions concurrently using the CTP hardware. The partitions are separated logically, each having a unique run number, but share the same trigger path up to the pre-scaling block. This feature is mainly intended to be used during commissioning and calibration runs. Only one of the partitions is interfaced to the higher level trigger and data acquisition system and can be used for physics data taking. The partitioning of the CTP requires a new control software architecture, which is described in the next section.
A schematic representation of the CTP is shown in figure 3. It is housed in a single 9U VME crate and consists of the following custom designed modules. The upgrades in the shutdown during 2013/2014 are highlighted: • CTP Machine Interface (CTPMI): receives the timing signals from the LHC and distributes them to the other modules through a custom build common backplane (COM bus).
• CTP Input (CTPIN): receives up to 124 trigger inputs over 4 cables, which are synchronized and aligned by each of the three CTPIN modules. Selected trigger signals are sent through the Pattern In Time (PIT) backplane to the CTPMON and CTPCORE modules. A firmware upgrade allows the transmission of the trigger signals on the PIT bus at double data rate (DDR), allowing for 320 transmitted trigger inputs in total.
• CTP Monitoring (CTPMON): performs bunch-by-bunch monitoring of the trigger signals on the PIT backplane.
• CTP Core (CTPCORE+): processes the input signals from sub-detectors and decides if a L1 Accept should be issued. 320 trigger input signals are received via the PIT backplane over 160 data lines operated at double data rate (80 MHz) and 192 trigger signals from direct electrical and optical inputs (also transmitted at DDR via 96 input lines) on the front panel of the CTPCORE+ module. This module has been redesigned and significantly upgraded. The number of input signals was increased from 160 to 512 in total, including the newly introduced direct electrical and optical front panel inputs. The optical inputs are foreseen to be used with future detector upgrades. The direct electrical inputs are currently used to connect to the topological processor, which needs a particularly low latency due to the additional processing time it uses. The computing capacity has been significantly increased by utilizing two Virtex 7 FPGAs. One is dedicated to the processing of the trigger signals, allowing for 512 logical combinations of the input signals (called trigger items) instead of 256, 16 instead of 8 bunch groups, random pre-scaling of trigger items and more random trigger generators. The second one is dedicated to monitoring tasks, allowing for 256 assignable bunch-by-bunch counters, in comparison to the 12 available before. Furthermore the trigger accept signal lines as well as the timing signals are triplicated to allow for the partitioning of the CTP. The CTPCORE+ also sends trigger summary information to the high level trigger (HLT) and the DAQ system.
• CTP Output (CTPOUT+): five modules distribute the trigger and timing signals via 25 cables to the sub-detectors. They also receive busy signals and calibration requests. The CTPOUT+ modules are redesigned to support the additional L1A and timing signals needed for the partitioning of the CTP.
• Common backplane (COM bus): distributes trigger and timing signals between the CTP modules. It has been upgraded to allow for five instead of four CTPOUT modules and provide the additional trigger and timing signals needed for the partitioning of the CTP.
During the shutdown in 2013/2014 CTPCORE and CTPOUT modules as well as the COM backplane have been replaced with their improved versions. The firmware of the CPTIN modules was upgraded to allow for the transmission of trigger signals at DDR on the PIT bus.

Software infrastructure
The CTP is a complex system essential for recording any physics data, hence it must not fail. Therefore the controlling software should be factorized into light-weight applications which are easy to maintain. The access to the hardware should be minimized, yet allowing for a sophisticated continuous monitoring. Finally, the running of three concurrent partitions should be supported. These considerations lead to the design of a completely new software architecture for the operation of the CTP, which is schematically shown in figure 4. Three independent core processes have access to the hardware. Following the aforementioned design ideas they only provide the minimal needed functionality to achieve maximal reliability: • CtpConfigurator: reads the hardware configuration from the trigger database and configures the hardware, the logic path as well as general settings, and ensures the correct configuration for a data taking session. To achieve this it provides a reservation system where each partition can request the CTP hardware. It then acts as a server for accessing the configuration and arbitrates the hardware access when configuring the CTP. A copy of the current configuration is also sent to an information server, where it is accessible to other applications e.g. for archiving purposes. Furthermore it handles pre-scales and bunch group updates during a run.
• CtpController: controls the CTP in the context of a run and is embedded in the ATLAS T/DAQ run control software framework. It will hold/resume the triggers, issue luminosity blocks, dynamically modify the masking of sub-detectors and steer the readout by the data acquisition system.
• CtpMonitor: will periodically read all available status information from the hardware and publish it on an information server.
• Monitoring clients: run in the context of a given partition. They read the monitoring information from the information server and provide trigger rates, busy fractions and status information per partition. They can also do complex data quality analysis and automated error detection. Any number of them can be deployed to perform the required actions on the monitoring information and publish results in a convenient way.

JINST 10 C02030
The implementation of this new software infrastructure required significant effort. Its operation, in particular when using multiple partitions, requires well defined configuration rules for the CTP and the software that must be strictly followed to ensure a smooth operation. In return a very flexible system is created that is easy to maintain due to the compartmentalization of tasks, provides sophisticated monitoring capabilities and can guarantee long term stable operation. Stable operation of the system is achieved by the modular design as crucial processes are separated from others, reducing the amount of code for the essential processes and hence the likelihood of crashes. Non-crucial processes can be restarted during data taking without interrupting an ongoing run, reducing the potential downtime of the experiment. As most processes, which do not require direct access to the hardware, are running on standard PCs a load balancing between several computers is possible, in contrast to the original software used during run 1. This allows for computing intensive analysis of the monitoring information by the monitoring clients.
The new software has been successfully tested in several weeks of detector commissioning in 2014 and proofed to perform as expected.