The sROD module for the ATLAS Tile Calorimeter Phase-II Upgrade Demonstrator

TileCal is the central hadronic calorimeter of the ATLAS experiment at the Large Hadron Collider (LHC) at CERN. The main upgrade of the LHC to increase the instantaneous luminosity is scheduled for 2022. The High Luminosity LHC, also called upgrade Phase-II, will imply a complete redesign of the read-out electronics in TileCal. In the new read-out architecture, the front-end electronics aims to transmit full digitized information to the back-end system in the counting rooms. Thus, the back-end system will also provide digital calibrated information with enhanced precision and granularity to the first level trigger to improve the trigger efficiencies. The demonstrator project is envisaged to qualify this new proposed architecture. A reduced part of the detector, 1/256 of the total, will be equipped with the new electronics during 2014 to evaluate the proposed architecture in real conditions. The upgraded Read-Out Driver (sROD) will be the core element of the back-end electronics in Phase-II. The sROD module is designed on a double mid-size AMC format and will operate under an AdvancedTCA framework. The module includes two Xilinx Series 7 Field Programmable Gate Arrays (FPGAs) for data receiving and processing, as well as the implementation of embedded systems. Related to optical connectors, the sROD uses 4 QSFPs to receive and transmit data from the front-end electronics and 1 Avago MiniPOD to send preprocessed data to the first level trigger system. An SFP module maintains the compatibility with the existing hardware. A complete description of the sROD module for the demonstrator including the main functionalities, circuit design and the control software and firmware will be presented.

. The ATLAS Trigger System. cells of the desired granularity. When the particles cross a cell, light is produced in the scintillating material. The intensity of this light is proportional to the energy deposited by the particle in the calorimeter. The produced light is collected using wave-length shifting fibers and conducted to photomultiplier tubes (PMTs) that convert it to an electrical signal.
TileCal is used to measure energy and position of hadrons, jets, taus and, jointly with other calorimeters, it provides the information on the missing transverse energy.

ATLAS TDAQ
The Trigger and Data Acquisition System (TDAQ) [5] is designed for event selection, processing and storage of the read-out data of the detector. The 40 MHz proton bunch crossings that take place in the interaction points can produce up to thousands of millions of particle events per second. This requires huge bandwidth and processing throughput that is neither available nor desirable. In order to cope with this, a mechanism is needed to select only interesting events from a physics point of view and, thus, decreasing event rate. This selection mechanism is based on three trigger levels in the data flow that define the different domains for the read-out electronics in terms of methods and rates for this selection. Figure 1 shows a diagram of the three trigger levels. Level 1 (L1) trigger information is obtained from the calorimeters and the muon chambers system. Signals produced in these systems every bunch crossing (at a rate of 40 MHz) are sent to the Central Trigger Processor (CTP), which takes the trigger decision upon possible interesting physics events. Level 1 trigger decreases the event rate down to 100 kHz. For every selected event, a Level 1 Accept (L1A) is propagated to the detector front-end electronics, where the event is captured and transmitted to the Read-Out Drivers -2 -(RODs) [12] first and stored on the Read Out Buffers (ROBs) later. The Level 2 (L2) trigger is a software trigger based on the Regions of Interest (ROI) defined by the Level 1 trigger. It accepts interesting events from ROBs and places them on the Event Builder decreasing the event rate down to 2 kHz. The Level 3 (L3) trigger, commonly named Event Filter (EF) uses more complex offline algorithms to reduce the event rate down to 200 Hz, before copying them in massive storage media. Figure 2 depicts a general view of the TileCal read-out chain. The first element in the front-end electronics is the PMT, which provide an electrical signal proportional to the light intensity on their front photo-cathode. This electrical signal is delivered to the 3-in-1 cards [6] where it is shaped and amplified using two gains. The output signals are sent to the Digitizer Boards [8], equipped with ADCs, TTCRx chips [9] and pipeline memories, as well as to the Adders [6], that make the analog sum of the signals of a tower of cells and send the result to the L1Calo, the calorimeter part of the L1 Trigger.

Front-end electronics
For each front-end module, the digitized data of the selected events in the Level 1 trigger system are transmitted to the Interface Board [7], where the data is packed, formatted and transmitted to the RODs in the back-end electronics using G-Link protocol [11] at 640 Mbps.

Back-end electronics
Each ROD board [12] receives information from eight consecutive TileCal modules with a total input bandwidth of 5.12 Gbps. The information of each data link is de-serialized using G-Link HDMP-1024 chips and routed through staging FPGAs to two Processing Units (PUs) [13]. The PU is the main processing core of the ROD. It is a mezzanine card that hosts two TMS320C6414 Digital Signal Processor (DSP) from Texas Instruments, which perform online (real time) algorithms [14] to obtain the reconstruction of the energy, the time and the computation of a quality factor of the reconstructed magnitudes for each front-end PMT. Other tasks performed by the DSPs are the synchronization of the data and trigger, compression of the digital samples, computation of the total transverse energy, digital error detection, busy signal generation and monitoring tasks. All this information is passed to the Output Controller FPGA, which packs it into the ROD Data Fragment. Finally, the data fragments are serialized and sent to the ROB using the VME Transition Modules (TMs). Other dedicated FPGAs in the ROD perform services like interfacing VME bus or communication with the TTC network.

TileCal Upgrade program 2.1 Motivation
The LHC has envisaged a series of upgrades in the machine in order to reach the design center of mass energy ( √ s = 14 TeV) and to increase the peak luminosity up to 5 times the nominal value (L peak = 5 × 10 34 cm −2 s −1 ). In order to adapt the detector to the new luminosity conditions, a complete replacement of the read-out electronics is foreseen in TileCal [15].

Upgrade of the read-out electronics architecture
After the completion of the different upgrade phases, the new installed architecture is aimed to provide: • Full digital L1 trigger.
• Complete detector data from the front-end to the back-end electronics every bunch-crossing.
• Redundancy at the level of data links to the back-end.
• Redundancy at the level of power supplies.
• Higher radiation tolerance. Figures 2 and 3 show the present and future read-out architectures respectively. The most remarkable difference is the boundary with the L1 selection. In the new architecture, the L1 trigger decision will be performed at ROD level, using digital data stored in pipeline memories. Together with the redundancy on the data links, the change of the L1 trigger boundary results in a considerable enlargement of the detector output band-width, as shown by table 1.

Upgrade schedule
During the foreseen detector long shutdowns different steps of the demonstrator program will be deployed. The ATLAS Upgrade program is divided in three different phases aligned with the scheduled LHC shutdowns. The practical execution of the TileCal upgrade will be carried out in the Phase-II, although some operations are scheduled for Phase-0 and Phase-I. Figure 4 shows the time line for the detector upgrade.
The first step in the upgrade program consists of the installation of a demonstrator of the new architecture shown in figure 3 in one of the 256 TileCal modules. This will be a hybrid module, in the sense that it will provide analog together with digital trigger information in order to be compatible with the old and the new systems. This hybrid drawer will be installed during the Phase-0 upgrade.
In the subsequent years three consecutive additional hybrid demonstrator modules will be installed, which will allow the evaluation of the new proposed architecture in a considerable area of the detector. These modules will work in the same way as the first one, providing digitized data every bunch crossing, analog trigger and digital trigger.
Finally, during the Phase-II long shutdown, the rest of the TileCal modules read-out architecture will be replaced by the new one, providing from this moment fully digital trigger.

Upgrade Demonstrator for Phase-0
In order to test the architecture shown in figure 3, several prototypes have to be developed. The target is to have a full slice of the future architecture fully operational and compatible with the present system. Figure 5 displays the block diagram of the Upgrade Demonstrator for Phase-0. Front-end electronics is divided into four mini-drawers, increasing the modularity to make its handling and insertion easier. Each minidrawer provides the infrastructure for reading 12 PMTs. A daughter board will transmit the serialized information via optical links to the back-end electronics. In the event of losing the link of one minidrawer, the other three keep operational. Besides, the analog signals from the old system adders will be sent to the L1Calo. The upgraded version of the ROD, the so-called super Read Out Driver (sROD), will receive the data from the four optical links to perform the signal reconstruction. Furthermore, it will handle the digital trigger information as well as the configuration of the front-end electronics.

sROD Demonstrator
The formerly introduced upgraded version of the ROD is needed in the future back-end electronics. For Phase-0 upgrade demonstrator, a prototype board of the sROD is being developed: the sROD Demonstrator. It will have to perform the read-out of a complete superdrawer, as well as the management of the L1 trigger information and configuration of the front-end electronics. It will also handle the Detector Control System (DCS) commands transmission and monitoring. For this purpose, the prototype requires: • Reception of digitized signals from 48 PMTs every 25 ns (40 MHz).
• Implementation of pipeline and de-randomizer memories.
• Implementation of signal reconstruction algorithms.
• Data transmission to the Read Out System (ROS).
• Control and configuration of the front-end electronics.
• Trigger data pre-processing and transmission to the L1Calo.
• DCS information management.

Board functionality
The sROD demonstrator prototype is intended to be operated either in a standalone mode or plugged as a mezzanine card in a carrier board inserted in an Advance Telecommunications Computing Architecture (ATCA) system. Thus, the prototype has a defined form-factor and backplane connector besides the front-panel connectivity.
The sROD demonstrator has to have the capability of reading one complete superdrawer. This functionality is achieved using state-of-the-art high speed optical connectors together with last generation high performance FPGAs equipped with Multi Gigabit Transceivers (MGTs). Figure 6 shows a block diagram depicting the functionality of the board.
-6 - Optical connectors. Starting from the I/O point of view, four QSFP connectors will provide the links with the four mini-drawers in a superdrawer. These connectors have four bidirectional transceivers capable to transmit data up to 10 Gbps per link, yielding a total of 40 Gbps input and 40 Gbps output bandwidth per connector. Each QSFP will receive four optical links: two with information of 12 PMTs of the minidrawer and two more with the redundant information of these 12 PMTs. Using this scheme, we assume that the information of 6 PMTs is digitized, packed, serialized and transmitted in each optical link.
In order to evaluate a different technology the sROD will also include an AVAGO MiniPOD receiver, that hosts 12links capable of running at 10 Gbps each. In this case, the sROD board could be connected to a front-end daughterboard equipped with MiniPOD transmitter connector. This adds 120 Gbps to the input bandwidth of the sROD demonstrator.
For backwards compatibility reasons a bidirectional SFP+ connector is also placed on the board. This way, the sROD will be capable of transmitting or receiving data from a ROD. The SFP+ provides 10 extra Gbps to the optical input and output of the board.
Finally, a transmitter MiniPOD will be used to send the pre-processed trigger information to the L1Calo. This connector adds 120 Gbps to the optical output bandwidth of the sROD.
Taking into account all the optical connectors, the sROD board has an (optical) input and output bandwith of 290 Gbps.
FPGAs. The core of the system are two Series 7 Xilinx FPGAs. The first one is a 1158-pin high performance Virtex-7 XC7VX485T with 485000 logic elements, 48 MGTs, 2800 DSP slices and 37080 kb of Block RAM (BRAM) memory. This FPGA will manage all the incoming data from the front-end since it is connected to the QSFP and the receiver MiniPOD. It will host the firmware needed to implement the decoding of the GigaBit Transceiver (GBT) protocol [16] used in the -7 -transmission from the front-end, the separation of data, TTC information and slow control from the GBT stream, the pipeline memories, the event selection based on the L1A, the data packing in the ROD format and the transmission of the ROD data fragments to the backplane using Gigabit Ethernet. For that purpose, it has dedicated pins for the backplane connector of the board.
The second is a Kintex-7 XC7K420T with 901 pins and 480000 logic elements, 24 MGTs, 1920 DSP slices and 34380 kb of BRAM. This FPGA is intended to receive data from the Virtex-7, to perform some trigger pre-processing tasks and transmit them to the L1Calo. In order to receive the data from the Virtex-7, a dedicated connection is routed between both FPGAs.
The reason of having two FPGAs was motivated on the fact that the sROD demonstrator board is a prototype which implements a reduced part of the final sROD architecture. In the final sROD design there will be several Virtex 7 to receive the data of many front-end modules and a separate Kintex 7 to collect pre-processed data from every channel and form bigger entities like towers or clusters. Thus, the functionality was desired to be split in two FPGAs already from the sROD demonstrator. This approach allows getting familiar from the beginning with the design organization, which determines some aspects that need to be studied, for example the latency of the data transfer from one FPGA to another, the transmission protocol to be selected or the interoperability between different kinds of MGTs present on the two different FPGAs.
Memory. The sROD board is equipped with two kinds of memory. Each FPGA is connected to a parallel flash chip in order to store configuration data, coefficients for DSP processing or a kernel image for the implementation of an embedded system based on MicroBlaze [17].
Besides the non-volatile memory, the FPGAs are connected to two 512 MB DDR3 chips for its use with the embedded system, in the case of using a standard embedded Linux distribution, which would require higher utilization of RAM memory.
Extra connectivity. The sROD prototype will include also an USB-UART. This device allows to easily establish a serial connection with the core of the embedded system within the FPGA.
A dedicated Ethernet connection is present on the board. This connector will permit Gigabit Ethernet communication with the sROD using a standard Ethernet cable when operated in standalone mode, or from an external computer when inserted to the ATCA system.
The Joint Test Action Group (JTAG) programming of the FPGA will be performed using a USB cable by means of a JTAG to USB [18] converter from Digilent. This little device allows the reduction of space and avoids the necessity to use the Xilinx JTAG Cable.
A Modular Management Controller (MMC) board facilitates the operation of the basic services to the prototype when connected to the ATCA system using the Intelligent Platform Management Interface (IPMI) [19] protocol. This device controls the switch-on or switch-off of the power supplies provided by the ATCA system to the sROD board for hot-plugging or unplugging, and monitors some relevant parameters like current consumption, board temperatures, etc.
In order to expand the functionality of the prototype, the sROD includes a FPGA Mezzanine Card (FMC) Connector, which is a standardized high speed connector, very present in the industry these days.
Power. The special power supply requirements for the FPGAs, the different optical devices and the DDR3 memories results in a complex design of the local voltage regulators needed for the -8 -sROD board. A total of 10 different voltage levels are required as well as special highly constrained filtering stages for the power integrity. Also there are many strict power booting specifications in the terms of time sequencing or monotonicity of the voltage supplies on the power on. For this reason, a considerable area of the board is populated with power devices.

ATCA framework
As introduced in section 2.4.1, the sROD demonstrator board is designed under the required specifications to be working under an ATCA equipment framework. The ATCA is an industry standard of modular electronics, that requires having a common infrastructure (chassis, power supply, backplane) and different modules (cards) that communicate among themselves using this infrastructure. All the equipments for the laboratory setup have been selected from the same manufacturer (Radisys), to avoid interoperability problems. The chassis is a SYS6000 platform, capable of hosting six ATCA cards (blades) in horizontal position connected to a dual-star topology backplane. A switch and shelf manager controls the data transfers between blades through the backplane. There is a dedicated computer module: the ATCA-4500. The computer has intuitive user access to the backplane of the ATCA system. An ATCA carrier blade (ATCA-1200) will house the sROD providing mechanical support and power supply distribution for mezzanine modules, as well as high-speed communication to the Rear Transition Module (RTM), to the backplane and between hosted modules.

Mechanical design
The sROD demonstrator board is compliant with a standard called Advance Mezzanine Card (AMC). The specific form factor is double mid-size AMC, with dimensions of 180.6 mm × 148.5 mm. The sROD demonstrator module can be plugged as a mezzanine card to an ATCA carrier blade or directly to the backplane in a uTCA crate, as well as operated in standalone mode as it was introduced on section 2.4.1. The AMC backplane connector has been integrated in the PCB for obtaining a more compact design.

Layout and PCB design
The layout of the sROD prototype is already finished and the board is in routing phase. The sROD includes more than 1200 components, and requires a high integration level on the PCB. The critical areas are the 400-pin FMC connector, the 170-pin AMC connector and the two high-density package FPGAs: the Kintex-7 with 900 pins and the Virtex-7 with 1556 pins. Six 144-pin Land Grid Array (LGA) packages are used for the DC/DC µModule regulators resulting in a highly integrated power section of the board.
The sROD stack-up has been thoroughly selected to fulfill the high-speed design specifications and constraints. As dielectric material between copper planes, the special NELCO 4000-13 SI has been chosen in order to obtain optimal values of dielectric constant (3.2 at 10 GHz) and dissipation factor (0.008 at 10 GHz). A total of 16 layers, divided in 8 power and grounding planes and 8 signal layers, are needed for the proper routing of the design. The PCB has a total thickness of 1.6 mm, which makes the sROD demonstrator AMC standard compliant. Figure 7 shows a picture of the sROD prototype after the layout and before starting the routing process.

Conclusions
The future upgrade of the LHC will bring an increase of the peak design luminosity by a factor of 5-7. Due to the aging and in order to cope with the increased rates produced by the upgraded LHC, the TileCal read-out will be replaced. The Tile Upgrade Demonstrator Program is being developed in order to test a slice of the TileCal future read-out electronics architecture. At the end of the present LHC shutdown (2013-2014) a hybrid TileCal front-end module will be installed on the detector. This module will be read-out by the sROD demonstrator, a PCB that deploys a fraction of the future sROD boards read-out capabilities as well as other functionalities such as pipe-lining and de-randomizing, trigger information pre-processing and transmission, front-end configuration or monitoring. The layout of the board is ready and the prototype is at this moment on routing phase.