Deployment of the CMS Tracker AMC as backend for the CMS pixel detector

The silicon pixel detector of the CMS experiment at CERN will be replaced with an upgraded version at the beginning of 2017 with the new detector featuring an additional barrel- and end-cap layer resulting in an increased number of fully digital read-out links running at 400 Mbps. New versions of the PSI46 Read-Out Chip and Token Bit Manager have been developed to operate at higher rates and reduce data loss. Front-End Controller and Front-End Driver boards, based on the μTCA compatible CMS Tracker AMC, a variant of the FC7 card, are being developed using different mezzanines to host the optical links for the digital read-out and control system. An overview of the system architecture is presented, with details on the implementation, and first results obtained from test systems.

The silicon pixel detector of the CMS experiment at CERN will be replaced with an upgraded version at the beginning of 2017 with the new detector featuring an additional barrel-and end-cap layer resulting in an increased number of fully digital read-out links running at 400 Mbps. New versions of the PSI46 Read-Out Chip and Token Bit Manager have been developed to operate at higher rates and reduce data loss. Front-End Controller and Front-End Driver boards, based on the µTCA compatible CMS Tracker AMC, a variant of the FC7 card, are being developed using different mezzanines to host the optical links for the digital read-out and control system. An overview of the system architecture is presented, with details on the implementation, and first results obtained from test systems.

K
: Detector control systems (detector and experiment monitoring and slow-control systems, architecture, hardware, algorithms, databases); Data acquisition concepts; Optical detector readout concepts; Modular electronics

Introduction
During the extended year-end technical stop 2016/2017, the Pixel Detector of the CMS Experiment is scheduled to be upgraded in order to cope with higher occupancy and data rates expected during the later phases of LHC operation. This so-called Phase 1 upgrade will add an additional layer in the forward-and barrel pixel detectors and thus significantly increase the number of required read-out links from 1120 to 1696 for the barrel-and from 448 to 672 in the forward-pixel system. The first and second layer of the barrel system will use four and two links per module respectively to cope with the high occupancy and thus data rate, which yields a total of 2368 readout links with almost 100% filling factor for the optical ribbons carrying the signals. In addition, the new pixel detector will feature a fully digital read-out system including new backend electronics.
For this upgrade, digital versions of the read-out chip (PSI46dig v.2.1) with increased buffering capabilities [1] and token-bit manager ASIC have been developed. Instead of encoding the signal in a 40 MHz analog data stream, the new, digital read-out chips (ROCs) operate on a 160 MHz clock and the data of two streams of eight ROCs each is multiplexed in a 320 Mbps stream by the token-bit manager (TBM). This stream is then encoded using a four-to-five bit scheme to reduce bit-errors during transmission [2]. The readout links will thus operate on a 400 Mbps digital protocol and will be able to handle the ≈ 300 Mbps maximum data rate expected under worst conditions with a margin of about 10%.
The output signal of the TBM is converted to an optical signal off the module and transmitted on a fiber to a Front-End Driver (FED) board in the backend electronics. In order to fit all the logic required for the decoding and deserialization of 48 input channels of a single FED card, the -1 -µTCA-based CMS Tracker AMC,1 a revised version of the FC7 [3] used in the CMS Trigger and Command Distribution System (TCDS) [4], has been selected as platform for the new digital FED. Since the µTCA standard offers several distinct advantages over the presently installed VME-based system, spare-parts are limited and components are no longer available, it is planned to also upgrade the backend part of the control system of the Phase 1 pixel detector to use the CTA as Front-End Controller (FEC).

The Phase 1 pixel system
The overall design of the proposed backend electronics for the Phase 1 pixel detector follows a µTCA schema adopted by CMS with some minor modifications to take into account the increased bandwidth requirements of the FEDs towards the central DAQ system in the Phase 1 pixel detector. It uses a redundant, dual-star µTCA backplane to distribute clock-, trigger-and fast commands that are received from the central TCDS system via a specifically developed module called AMC13 [5]. A software library called µHAL and IPBUS, the corresponding firmware IP block, are used for control of the AMCs [6].   Figure 1 shows an overview of the system architecture including auxiliary components required to interface with the central CMS services. The pixel data from the detector front-end is sent to the FED, which decodes the incoming data stream and assembles the channel data of all 48 input channels into event fragments that are then pushed to the central DAQ.
The control part of the system requires two different flavors of FECs (compare figure 1) that are based on the CTA and use identical hardware: • Pixel FEC: this device distributes clock, trigger and fast signals to the pixel modules and in addition is responsible for controlling the DAC registers of ROCs and TBM.
• Tracker FEC: This type of FEC is used to program components in the pixel supply-electronics like opto-hybrids and DC-DC converters via the I2C interface of a CCU2 [7].
Overall, a total of 72 AMCs, with 56 being FEDs, 14 Pixel-FECs and 2 Tracker-FECs, distributed over a total of 7 crates, will be required to control and read-out the barrel-and forward pixel detector.

The CMS Tracker AMC (CTA)
The CTA shown in figure 2, which is a variant of the FC7 card [3], is a full-size, double-width Advanced Mezzanine card holding a Xilinx Kintex 7 FPGA and offering two low-pin-count compatible (LPC) FPGA Mezzanine Card (FMC) slots. These slots allow the use of a variety of mezzanines for various applications, including compatibility with multi-Gigabit optical transceivers. The total number of high-speed (10 Gbps) serial links to the FPGA available on the front-panel is 20 and 12 links are available on the backplane connector. Furthermore, there is a block of 4 Gb DDR3 RAM for data buffering that supports a transfer rate of 30 Gbps, and the firmware can be uploaded to the FPGA via a microSD card. A micro-controller compatible with AMC specifications is also included for management and monitoring of the card. The application specific FMCs and firmware that make a CTA into a FED or FEC, respectively, are described in detail in sections 3 and 4.

The Phase 1 digital front-end driver
As mentioned in section 1, the data from the detector modules will be transmitted to the backend via a 400 Mbps digital protocol. Since this already includes four-to-five bit encoding, the real bandwidth available for pixel data is 320 Mbps which is sufficient to accommodate data rates expected from modules closest to the interaction point. However, the output bandwidth to the central DAQ system is limited and thus links from inner-and outer detector layers can be mixed to balance the load. Given the many changes, it was decided to replace the present system by a µTCA Front-End Driver board based on the CMS Tracker AMC that can handle more input links and higher output data rates than the VME parts. The hardware of a Pixel FED consists of a CTA board with one or two Receiver-FMCs (Rx-FMC), which are mezzanines having two 12-channel optical receivers, collecting signals from the pixel detector, and one SFP+ 10 Gbps S-Link Express transmitter for data transmission to the central DAQ. One physical CTA FED, which consists of two logical FEDs (represented y the two FMCs), can read out 48 channels and transmit output data with 20 Gbps, yielding a margin of about 50% over what is expected from simulation. The receivers used on the FMC are manufactured by FITEL and were specifically adapted to the wavelength at which the pixel-opto-hybrid (POH) transmitters operate (1310 nm) and tested at the desired speed of these links. Figure 3 shows a picture of a CTA equipped with a single Rx-FMC (a) and results of bit-error rate (BER) versus optical modulation amplitude (OMA) measurements done with a pre-production FMC (b). The specifications for the minimal output of the transmitter and minimum sensitivity of the receiver are shown for reference. In order to synchronously read out the module data, the Pixel FED also needs to receive clock-and trigger signals from the TCDS system and the AMC13 via the µTCA backplane. In addition, a trigger-throttle signal (TTS) is sent back to the AMC13 which forwards this signal to the trigger system.   The entire FED firmware is running on the single Kintex 7 FPGA of the CTA but is divided into several logical blocks. A frontend block, implemented for each of the 48 input channels, is responsible for finding the correct phase of the 400 Mbps incoming data stream, and after correctly identifying the individual frames, decoding and demultiplexing of the data into individual streams for each TBM core. Then TBM-and ROC headers, pixel data and the frame trailer are identified and the consistency of the incoming data is checked. A dedicated finite-state machine forwards eventual errors to the backend IP block. A spy buffer at this stage allows inspection of the data for each TBM core separately, before it is pushed to the FIFOs of the backend IP.
The backend IP block, in addition to handling the TCDS signals including the TTS status, serially drains the decoded TBM data for each channel into a global FIFO that contains a complete event fragment from all 48 input channels. Then the data is synchronously pushed to the central DAQ via the S-Link Express 10 Gbps link. In addition, a local mode is implemented to read event data via IPBUS and the Gigabit-Ethernet link of the CTA. This local mode is mainly used for debugging and calibration during commissioning of the detector. Figure 4 shows a schematic overview of the frontend deserializer block and the interface to the backend IP block. The second diagram shows the handling of event fragments in the latter.
An implementation of a single frontend deserializer block has successfully been tested on the CTA board with very promising results. Next, a version for handling all 12 channels of a single FITEL receiver will be integrated with the backend IP block to form a prototype FED that can be used for further development and lab testing.

The front-end controller
Due to the increased number of detector modules and thus control links in the Phase 1 pixel detector the present control system has to be expanded, and since spare FECs and mezzanines are limited, it is foreseen to use µTCA components to replace the entire backend part of the control system. As mentioned in section 2, two kinds of Front-End Controller boards, that are functionally equivalent -5 -to the present VME parts, are required for the operation of the CMS pixel detector. Both devices use identical hardware but differ in the firmware running on the FPGA. The following sections will present an overview of the mezzanine cards (FMCs) and the specific communication protocols used.

Tracker FEC
The so-called Tracker FEC or CCS [8] was initially developed to control the devices used in the CMS Silicon-Strip Tracker via a CCU chip and to distribute clock and trigger signals to the detector frontend. It was adapted for the pixel system as it offers a way to program auxiliary pixel supply electronics, via the CCU chips' I2C-and PIA interfaces, that is completely independent from the control of the pixel modules. However, the distribution of clock and trigger signals via the Tracker FEC is not used for the pixel detector. Each Tracker FEC can connect a number of CCU chips in a ring-like topology via redundant optical fiber connections that carry clock and data signals and the control is done via a token-ring protocol. Two so-called DOH3 transceivers convert the signal from optical-to electrical levels and forward it to the CCUs. The µTCA implementation of the Tracker FEC will also use the CTA board and in order to archive compatibility with the fibers and transceivers already installed in the detector, an FMC was adapted from the TCDS project that uses commercial 1 Gbps SFP units, that can run at the lower speed (80 Mbps) of the control links, to connect to the DOHs. Figure 5 shows a CTA board with two SFP FMCs installed. This µTCA Tracker FEC can drive a total of four redundant CCU control-rings via the 16 transceivers of the two FMCs, which is exactly the number of rings required for the forward-and barrel-pixel systems so that no more than two boards (plus some spares) are required for the whole Phase 1 pixel detector.
The firmware written for this Tracker FEC is loosely based on the logic used in the VME system, but had to be restructured to run on the Kintex 7 of the CTA. Notably, the control and status registers that are exposed to the software have been kept the same, but the internal part controlling tokens and data in the ring has been adapted and restructured. As a direct consequence, the FEC 3Digital Opto Hybrid.
-6 -control software only had to undergo minor adaptations at the low level hardware-access calls to use the µHAL library.

Pixel FEC
In terms of hardware, the CMS Pixel Front-End Controller, or Pixel FEC, is completely identical to the Tracker FEC presented in the previous section. However, the underlying communication protocol and thus firmware differ. The Pixel FEC is responsible for distributing clock, trigger and fast signals to the detector frontend and for programming the DAC registers of the TBM and ROC chips on the detector modules. For this purpose, an I2C-like communication protocol, running at 80 Mbps on dedicated clock and data lines was developed. Commands are being sent to the modules via optical fibers through a modified version of the DOH (mDOH) mounted on the supply electronics. This connection, however, is not redundant. The clock and fast signals like trigger, reset and calibration pulse request, that are encoded in the clock line, are decoded by a TPLL-chip4 after the opto-electrical conversion, and forwarded to the modules on dedicated lines. The DAC programming commands for ROC and TBM chips are carried on the data line of the control link and are routed to the frontend through a series of delay and addressing ASICs. Each module connected to a pixel-control link is identified by a unique, hardwired four-bit hub address. As with the Tracker FEC described in the last section, the number of available optical in-/outputs per board is reduced from 64 on the VME part to 32, and 8 mDOHs can be connected. Due to the increased number of required control links, especially in the forward-pixel system, a total of 14 µTCA Pixel FECs will be required for the Phase 1 detector.
Development of the Pixel FEC firmware is progressing, with the implementation of the communication protocol for ROC and TBM programming and the TTC decoder completed. IP blocks developed for the Tracker FEC can be reused for the handling and control of the optical components mounted on the FMC, once the development moves from an electrically connected test setup to the full optical pixel control chain.

System tests
In order to test and validate the Phase 1 system under realistic operating conditions, a pilot system has been installed inside CMS with the present forward-pixel detector during Long Shutdown 1. This pilot system uses eight pre-production digital pixel modules, prototypes of the new forward pixel service electronics and optical links. Available VME parts are used as control system and a VME-based Pixel FED has been adapted to the digital readout protocol. This pilot system is fully integrated with the central CMS services, including dedicated TCDS and central DAQ partitions. As a second step it is planned to replace the VME parts with the µTCA pre-production components. Furthermore, several larger-scale test systems will be set up before the installation of the Phase 1 detector during the extended year-end technical stop 2016/2017. These will allow to test the components and system architecture in a more realistic configuration and ensure scalability of the system. Important milestones that have to be covered include: 4Tracker Phase-Locked Loop.

JINST 11 C01056
• Forward-pixel half-disk test stand with approximately 50 read-out channels. This corresponds to the number of input channels of a single µTCA Pixel FED and can be used to validate the FED hardware and check if all input channels are working correctly.
• Forward-pixel half-cylinder test stand. Forward-pixel parts are manufactured and assembled in the United States and their functionality needs to be verified after shipment to CERN. This will require, both at FNAL and CERN, a system with four FEDs, two Pixel FECs and a single Tracker FEC and will allow to test the interplay of control and read-out system in a fairly large system.
• In addition, a similar-sized system will be used to participate in global runs with the central DAQ system in order to verify that the Phase 1 pixel DAQ integrates well with the rest of CMS.

Conclusions
In order to adapt to the 400 Mbps digital read-out scheme of the CMS Phase 1 pixel detector and the greater number of read-out and control links, new Front-end Driver and Front-end Controller cards have been developed based on the µTCA compliant CMS Tracker AMC. Pre-production parts of all components are available and are being used for firmware development and test-systems. A series of system-level tests of growing scale has begun and will continue up to the installation of the new pixel detector in CMS during the extended year-end technical stop in 2016/2017.