A new data acquisition system for the CMS Phase 1 pixel detector

A new pixel detector will be installed in the CMS experiment during the extended technical stop of the LHC at the beginning of 2017. The new pixel detector, built from four layers in the barrel region and three layers on each end of the forward region, is equipped with upgraded front-end readout electronics, specifically designed to handle the high particle hit rates created in the LHC environment. The DAQ back-end was entirely redesigned to handle the increased number of readout channels, the higher data rates per channel and the new digital data format. Based entirely on the microTCA standard, new front-end controller (FEC) and front-end driver (FED) cards have been developed, prototyped and produced with custom optical link mezzanines mounted on the FC7 AMC and custom firmware. At the same time as the new detector is being assembled, the DAQ system is set up and its integration into the CMS central DAQ system tested by running the pilot blade detector already installed in CMS. This work describes the DAQ system, integration tests and gives an outline for the activities up to commissioning the final system at CMS in 2017.

A : A new pixel detector will be installed in the CMS experiment during the extended technical stop of the LHC at the beginning of 2017. The new pixel detector, built from four layers in the barrel region and three layers on each end of the forward region, is equipped with upgraded front-end readout electronics, specifically designed to handle the high particle hit rates created in the LHC environment. The DAQ back-end was entirely redesigned to handle the increased number of readout channels, the higher data rates per channel and the new digital data format. Based entirely on the microTCA standard, new front-end controller (FEC) and front-end driver (FED) cards have been developed, prototyped and produced with custom optical link mezzanines mounted on the FC7 AMC and custom firmware. At the same time as the new detector is being assembled, the DAQ system is set up and its integration into the CMS central DAQ system tested by running the pilot blade detector already installed in CMS. This work describes the DAQ system, integration tests and gives an outline for the activities up to commissioning the final system at CMS in 2017.

K
: Data acquisition concepts; Detector control systems (detector and experiment monitoring and slow-control systems, architecture, hardware, algorithms, databases)

Introduction
The innermost part of the CMS experiment hosted at CERN, Switzerland, is built from a detector based on the silicon hybrid pixel technology. This pixel detector consists of three layers in the barrel shaped part, commonly named BPIX, and two disks on each end of CMS in the forward part, called FPIX. The n-in-n silicon sensors are of a thickness of 285 µm. BPIX and FPIX together possess a total of 66M pixels, with each having an active area of 150 µm × 100 µm.
The existing CMS pixel detector was designed to operate at an instantaneous luminosity L = 10 34 cm −2 s −1 and a bunch spacing of 25ns [1]. The assumed average number of inelastic collisions (pile-up) per bunch crossing under these conditions was 25. Due to the excellent performance of the LHC these values have been reached and already exceeded by more than 20%. Towards the end of the second running period of the LHC in 2016 the instantaneous luminosity reached values of 1.3-1.5 ×10 34 cm −2 s −1 .
Limitations in the readout logic of the pixel detector front-end parts cause dynamic hit detection inefficiencies. At the design luminosity these inefficiencies amount to a random data loss of 2% for the innermost BPIX layer [2]. The inefficiencies will increase even further as the LHC reaches higher instantaneous luminosities causing a substantial performance loss in the reconstruction of particle tracks and thus the physics performance of the whole experiment. In order to maintain the excellent performance of the CMS experiment it is planned to install a new pixel detector during an extended year-end technical stop (EYETS). This detector will have an upgraded front-end readout chip, a larger number of layers in the barrel and the forward region, and a new data acquisition system (DAQ) in the CMS service cavern.
Following an overview of the new and improved Phase 1 CMS pixel detector this paper will focus on the design, layout, implementation and test of the DAQ system.

The CMS Phase 1 Upgrade
The Phase 1 Upgrade of the CMS pixel detector is an evolutionary improvement of the currently installed pixel detector. The approach of using silicon hybrid pixel detectors remains unchanged. Silicon sensors of the n-in-n type are again used as detector material but performance improvements have been made to the front-end electronics, namely the readout chips (ROCs) and the token bit manager (TBM), a chip to control all 16 ROCs on a single module, to cope with the higher particle hit rates. Both the BPIX and FPIX parts of the detector are extended by an additional layer, with the innermost layer of the BPIX part moving closer to the interaction point and the outermost layer moving further away, covering currently unused space in the detector volume. A comparison of the old and new detector layout is shown in figure 1. The FPIX detector part will be equipped with a third disk on each end of CMS. In layer two to four of the BPIX part and all FPIX disks, the PSI46digV2 [3] readout chip will be used. This ROC possesses enlarged data and time stamp buffers compared to its predecessor, the PSI46v2. The PSI46digV2 chip is designed to handle hit rates of up to 120 MHz/cm 2 while keeping the hit inefficiency below 2%. These conditions are expected for the second layer of the barrel region, with lower hit rates expected on later three and four, and in the forward regions. Closest to the interaction point, in layer 1 of the BPIX, the newly developed PROC600 [4] readout chip will be used. To cope with hit rates of up to 600 MHz/cm 2 a new column drain mechanism based on 2x2 pixel clusters has been implemented. The data format in which the ROCs transmit their collected hit information has been changed from a 6-bit analog level encoding to a digital data transmission protocol in order to increase the transmission speed to 320 MBit/s. The usage of low weight materials for detector support structures and the switch to a two-phase evaporative CO 2 cooling system leads to a significant reduction of nonsensitive material in the detector volume. A comparison of the material budget for the old and new detector can be seen in figure 1(b). For the new detector most of the material has been moved to the very high eta region. A detailed report on the CMS Pixel Phase 1 detector is given in [5].

System requirements and readout architecture of the new DAQ system
The change from analogue to digital readout made it necessary to also upgrade the entire readout chain up to the back-end readout electronics. The readout chips transmit their data on a digital data line clocked at a speed of 160 MHz. Two serial data lines of up to eight ROCs are multiplexed to a 320 MHz data stream by the Token Bit Manager (TBM) chip. In order to balance the readout links during data transmission, a four-to-five bit encoding is applied, resulting in a 400 Mbps serial data stream coming from each TBM. For the first layer of the CMS pixel detector the data rates are estimated to reach 600 MHz/cm 2 , thus requiring for two TBMs and two optical fibres for each module of 16 ROCs. In figure 2 the full readout architecture is shown. The 400 Mbps data stream from the TBMs is converted to an optical signal in the Pixel Opto-Hybrids (POH) and transmitted to the Front-End Drivers (FED). Together with the Front-End Controllers (FEC) that send control signals to the detector front-end and slow control commands to the readout periphery, the new DAQ system is entirely based on the FC7 card, a full size, double width µTCA AMC. This card is used by several CMS subsystems like the CMS Trigger and Command Distribution System (TCDS), the CMS-TOTEM Precision Proton Spectrometer (CTPPS), the Hadron Calorimeter and on work for R&D activities [6].

Pixel FEDs and FECs
All FC7 boards share the same basic hardware configuration. The cards are equipped with a Kintex 7 FPGA, 4Gb of DDR3 RAM for data buffering and twelve 10 Gbps high speed links to the crate back plane. Additionally each FC7 can house up to two FPGA Mezzanine Cards (FMC) -3 -that can be customised to a specific use case as shown in figures 3(a) and 3(b). In the CMS Pixel Phase 1 system, these FMCs are used to enable the FEDs and FECs to perform their specific tasks, interfacing to the dedicated optical links for readout and control.
The FEDs are equipped with a single FMC card. The FED FMC contains two optical receivers1 manufactured by the company FITEL, with twelve inputs each, shown in figure 3(c). The FITEL semi-conductor parts are optimised for a wave length of 1310 nm and compatible with the 400 Mbps serial data stream coming from the detector front-end. Additionally each FED FMC houses a cage to place either a 5 Gbps or 10 Gbps optical Ethernet SFP+ link, over which the collected event data is transferred to the CMS Central DAQ system. The FECs come in two different flavours: Tracker FEC and Pixel FEC. The Tracker FEC is used to program the supply electronics like the POH, used for electrical to optical signal conversion, or the DC-DC power converters via the I 2 C interface of a CCU [7]. The Pixel FEC on the other hand is used to supply the detector front-end chips with a 40 MHz clock, distribute trigger and other fast control signals to the front-end and to program the DAC settings on TBM and ROCs. Both Pixel and Tracker FEC use the same FMC hardware as shown in figure 3(d) and only differ by the firmware loaded to the FPGA.

The full DAQ backend layout
The CMS pixel detector readout and control system consists of twelve µTCA crates, four of which are used to house FEDs and FECs for FPIX, while the other eight hold the BPIX backend electronics. All crates2 manufactured by Schroff are based on standard units used in telecommunication with a backplane made to CMS specifications. They can hold up to twelve full-size AMCs, are equipped with two fully redundant power supplies and have space for a 13 th half-size AMC where the AMC13 card [8] is placed. This extra AMC is used to receive and distribute CMS clock and trigger signals within the crate. Through the µTCA Carrier Hub (MCH) the crate and all AMC cards in it are connected to standard 1Gbps Ethernet. The special backplane configuration has twelve high speed links between the AMC slots and the AMC13.
The crates for BPIX and FPIX are separated into three different racks. Each of the FPIX crates holds seven FEDs and two Pixel FECs. For the BPIX detector which has more channels than FPIX each crate is equipped with 10 FEDs and a single Pixel FEC. For both the FPIX and BPIX, a single Tracker FEC each is enough to control the front-end periphery electronics. The layout of the DAQ backend is shown in figure 4. As seen in the layout, the bottom of the racks is reserved for optical patch panels, where the optical fibres from the detector front-end arrive. If needed in the future, the fibre connections can be remapped to balance the data input rates into the FEDs.

System tests
Several test stands and development benches based on the µTCA readout system at different detector assembly centres have been created, where hardware parts, firmware for the FED and FECs, as well as the Pixel Online Software (POS), are being evaluated, systematically studied and used to verify the overall integration of all detector parts. As an example three centres are mentioned:  • Uni Zurich, as part of the BPIX community, is responsible for testing the optical components on the service cylinders for the barrel part of the CMS Pixel detector. While tests after assembly of the BPIX are made using the legacy VME system and custom built electronics, the optical parts are also checked using the new µTCA readout system. Additionally the overall integration of the new readout and control system with the BPIX front-end and service electronics is being verified.

ID Units
Having several data acquisition systems distributed to different centres and dedicated to different tasks have advanced the integration of the POS software package with the detector firmware and the front-end and has accelerated the learning of the FPIX and BPIX user community. At the same time new features in the FED and FEC firmware could be tested without losing time during detector assembly. Finally the in-situ use of the CMS Pixel readout system has made it possible to spot firmware and software bugs before the final integration of the full system in the experiment.

Soak test
In order to minimise problems during this startup phase of the CMS pixel detector in 2017 a 'soak test' is performed with all DAQ back-end parts. Twelve µTCA crates, the same crates that later will be installed in the CMS service cavern, are loaded with up to twelve FED readout cards. To create a high load on the FPGA the FEDs are operated running in an emulated data mode, where the FED firmware creates an event for each L1A trigger it receives and fills it with a fixed pattern of eight hits per TBM readout channel. At the same time all FEDs and crates are constantly monitored. Temperatures, operation voltages and currents on all FC7 cards and FED FMCs are monitored and logged, as well as the load on the power supplies. This test is run for several hundred hours before the equipment is installed in the CMS experiment to gain experience with the full-size system and to ensure that 'infant mortality' of newly assembled cards will not hinder the commissioning of the new pixel detector in the CMS experiment.

Summary and outlook
A new DAQ system based on the µTCA standard has been designed and is currently under development to accommodate for the new digital readout scheme and higher data rates coming from the front-end. New Front-End Drivers and Front-End Controllers based on the FC7 board have been designed and produced. Several test stands have been set up to facilitate development and debugging of FED and FEC firmware and also to test the system integration of the DAQ backend, the different detector front-end configurations for BPIX and FPIX and the overall software framework. A soak test for all backend electronics has been set up to test the hardware under realistic load conditions before it is installed in the service cavern of the CMS experiment.