Firmware development and testing of the ATLAS Pixel Detector / IBL ROD card

The ATLAS Experiment is reworking and upgrading systems during the current LHC shut down. In particular, the Pixel detector has inserted an additional inner layer called the Insertable B-Layer (IBL). The Readout-Driver card (ROD), the Back-of-Crate card (BOC), and the S-Link together form the essential frontend data path of the IBL's off-detector DAQ system. The strategy for IBL ROD firmware development was three-fold: keeping as much of the Pixel ROD datapath firmware logic as possible, employing a complete new scheme of steering and calibration firmware, and designing the overall system to prepare for a future unified code version integrating IBL and Pixel layers. Essential features such as data formatting, frontend-specific error handling, and calibration are added to the ROD data path. An IBL DAQ test bench using a realistic front-end chip model was created to serve as an initial framework for full offline electronic system simulation. In this document, major firmware achievements concerning the IBL ROD data path implementation, test on the test bench and ROD prototypes, will be reported. Recent Pixel collaboration efforts focus on finalizing hardware and firmware tests for the IBL. The plan is to approach a complete IBL DAQ hardware-software installation by the end of 2014.


Introduction
The ATLAS experiment at the LHC [1] planned to upgrade the existing Pixel Detector [2,3] with the insertion of an innermost silicon layer, called the Insertable B-layer (IBL) [4]. The project has been designed in order to increase the tracking robustness against failures as well as to improve the measurement precision even at the higher LHC luminosities. The IBL has been installed during the current LHC shutdown.
The IBL read-out electronics have been redesigned to accomplish the enhanced detector performance. A new front-end ASIC, called the FE-I4 [5] has been designed to handle the larger occupancy as well as to manage the increased bandwidth expected for the IBL. Figure 1 shows the new off-detector electronics [6] that have been designed as well in order to overcome limitations in the current system; they consist of two 9U-VME cards: the Back-of-Crate (BOC) and Read-Out Driver (ROD), which respectively implement the optical I/O interfaces and data processing [7]. Each card pair processes data received from 32 FE-I4 data links, for a total I/O bandwidth of 5.12 Gb/s. Moreover, the new ROD provides enhanced performance in the calibration runs. These are conducted by injecting test charges into the single pixel preamplifier; these injections are iterated over several acquisition scans with different settings of the front-end electronics parameters. In the current version of the readout electronics, scan results are acquired over the VME bus, with a transfer rate limit of 7 MB/s. The new ROD design implements dedicated Gigabit Ethernet connections, thus enhancing the overall acquisition performance of the calibration runs. A new off-detector design was also motivated by the obsolescence of components used in the current cards.
The ROD card has been designed and fabricated on a 14-layer PCB. First prototypes were received from the producer in September 2011. The final version, namely revision D, was delivered starting in the summer of 2013. This last revision features an enhanced termination technique in all the receiving data lines from the BOC to the ROD and some special layout arrangements for critical lines and components. However, the main functions of the revisions have been kept backward-compatible so that the firmware under development works for all the ROD versions. Figure 2 shows the revision D ROD card; 20 boards (including 6 spares) have been produced for IBL. These boards have been intensively tested in different laboratories to debug the hardware and the firmware.

The ROD card
The ROD design features modern FPGA Xilinx devices: one Virtex-5 master device for control and two Spartan-6 slaves dedicated to data processing (gathering of front-end output, event building, and calibration data processing). For backward compatibility and reliability, two independent microprocessors are foreseen: a Digital Signal Processor (DSP) and a PowerPC embedded in the Virtex-5 [8,9].
In principle the two devices, the DSP and the Virtex-5 can carry out the same task of board control. However, the firmware development team has concentrated the efforts from the beginning of the project to use the Virtex-5 because it was clear that this component would have the power to extend the DSP features. In particular we have adapted the DSP original code, from the Pixel Detector to be directly used by the PowerPC embedded into the Virtex-5. Thus, the DSP device was left only as a backup solution for IBL. For the Layer 2 readout upgrade planned in 2015, in fact, this DSP component has not even been soldered into the ROD cards. Moreover, the Virtex-5 device was properly chosen since the component has an embedded hardware processor (PowerPC) that replaces the old DSP functionality and, in addition, allows other standard FPGA capabilities. Also, the other FPGAs chosen for this IBL ROD card have been selected as a compromise to have modern programmable devices but, at the same time, save as much as possible of the old VHDL firmware. The choice to keep a VME-based system was led by the necessity to install the new IBL layer together with the other pixel layers, using the same ATLAS DAQ infrastructure.
The ROD is a standard 9U VME64X board, and VME connectors are used to distribute the power to the card from the crate. Moreover, the VME bus interfaces the ROD with the DAQ controller system, acting as a backup to the main control link (Gigabit Ethernet) and allowing for remote upload of the FPGA's firmware. Other ROD external ports are: a Gigabit Ethernet towards the DAQ controller system (main connection), two Gigabit Ethernet ports to deliver the calibration scan results, one TTCrq mezzanine (port interface) to receive ATLAS clock and trigger commands, and four custom buses to the BOC through VME connectors. In more detail, ROD firmware is mainly divided in to master and slave parts. Figure 3 shows the main blocks of the ROD master -2 -firmware, which is implemented on the Virtex-5 and handles the ROD control path. It generates and sends event information to the slave FPGAs and it handles communications with the embedded PowerPC. The ROD slave firmware, which handles the ROD data path, is implemented on the two Spartan-6 slave FPGAs. The slave firmware processes incoming detector data in real time, handles error checking/tagging, and stores formatted data. Depending on the ROD operating mode, stored detector data will either be sent off-site or used in calibration runs. At the prototyping level, Pixel ROD firmware has been upgraded to suit the operations of the IBL. Upgrades to the Pixel ROD firmware include the elimination of functional blocks no longer needed by IBL, the restructuring of the code hierarchy, and the addition of new code blocks and algorithms. Static and dynamic memory components are hosted on the ROD; in particular, the Virtex-5 is equipped with a SO-DIMM DDR module. It is also supplied with a 64 Mbit Atmel Flash device devoted to the storage of both non-volatile parameters (e.g. Ethernet IP addresses) and software programs to be executed by the PowerPC. Figure 3 shows the main blocks for the ROD master firmware that is inside the Virtex5. Some parts are designed as embedded software (pink blocks) and linked to the main firmware designed in VHDL. They can be simulated together offline. This master code for the Virtex5 device interfaces with the Program Reset Manager (PRM) that is another FPGA used to control the VME bus. In addition, the Virtex5 uses an external dynamic memory (DDR) device and is connected to the BOC card via the so-called ROD bus. Also, the PowerPC (PPC) design is made of embedded software that controls and executes the control of the entire board. VHDL blocks in the figure have the task to readout the front-end chips and format the event data.

ROD firmware tests for the Pixel Detector
We have prepared a set of hardware and firmware tests to accelerate the validation of the ROD cards. The tests have been designed and grouped in the following sets which depend on the main functions and activities of the card: -firmware upload from VME, JTAG ports, -BOC-2-ROD dataflow, -ROD-2-TIM communication, -R/W tests for Virtex5 and Spartan6 on-board memory modules, -dataflow tests on the 3 Gb/s ports In particular the ROD firmware upload includes a configuration via VME of the firmware and software for embedded processors. The BOC-2-ROD communication checks the clock-phase, the I/O terminations, the predefined pattern sent to the ROD, and the cross-check of (SSTL3 I that is a standard of electric signals, in particular it stands for 3.3V Stub Series Terminated Logic) signals received from the BOC card. In addition to the above tests, the manufacturer carries out electrical validation tests on the cards just after the fabrication of the PCB and the population of components. X-ray tests are also performed on the BGA soldering of the main FPGAs.
For ATLAS Pixel Detector Layer 2, which requires 26 additional pairs of BOC-ROD cards, we have carried out a new production of the IBL cards for further boards.
In fact, the IBL boards can easily expand the bandwidth of Pixel Layer 2 as long as we provide dedicated firmware to interface with the present detector and, in particular, with the FEI3 chips instead of the IBL FEI4s. For Layer 2, including the spare cards used also for IBL, 40 RODs have -4 -  been fabricated in 2014 and will be installed in early 2015, just after the IBL commissioning. The test scheme shown in figure 4 refers to a complete data taken from 32 front-end channels. Here the data were generated on the BOC-ROD pair and the formatted events were sent back to the BOC and eventually to the 4 optical S-Links. The tests at CERN permitted a data taking with 2 real FEI4 (with 6 multiplexed copies) on 8 channels (1 SLink) up to 200 KHz. The Trigger-and-Control Interface Module (TIM) card (inside the crate) provides a trigger rate at 200 KHz (test with a TIM-BOC-ROD, ROBIN card + PC at CERN).
-5 - Figure 5 shows a full digital scan composed of 100 triggers, with a mask scheme 1 out of 8. In this way we were able to mask entire lines of pixels (links) again to prove the complete control of the system. Figure 6, eventually, shows a scan on the threshold carried out on the entire set of pixels read out by one pair of BOC-ROD cards.

Conclusion
The 14 RODs for the 14 staves of IBL have already been delivered to CERN and are currently running in USA15. Further spare boards have also been delivered to CERN and are undergoing system tests along with the other layers of the Pixel Detector. The ROD firmware is done, each board can interface with 32 FEI4 chips, and data-taking and calibration also work properly. The entire software-firmware system debugging is ongoing so that the ROD code is continuously under development for fine tuning. The IBL ROD card is also going to be used for the Pixel Layer 2 readout upgrade, by the end of 2014. In particular 26 boards will be used for the Layer 2 of the ATLAS Pixel Detector, and the IBL firmware will be slightly adapted to interface with FEI3 chips instead of FEI4 ones.