SPIDR, a general-purpose readout system for pixel ASICs

The SPIDR (Speedy PIxel Detector Readout) system is a flexible general-purpose readout platform that can be easily adapted to test and characterize new and existing detector readout ASICs. It is originally designed for the readout of pixel ASICs from the Medipix/Timepix family, but other types of ASICs or front-end circuits can be read out as well. The SPIDR system consists of an FPGA board with memory and various communication interfaces, FPGA firmware, CPU subsystem and an API library on the PC . The FPGA firmware can be adapted to read out other ASICs by re-using IP blocks. The available IP blocks include a UDP packet builder, 1 and 10 Gigabit Ethernet MAC's and a “soft core” CPU . Currently the firmware is targeted at the Xilinx VC707 development board and at a custom board called Compact-SPIDR . The firmware can easily be ported to other Xilinx 7 series and ultra scale FPGAs. The gap between an ASIC and the data acquisition back-end is bridged by the SPIDR system. Using the high pin count VITA 57 FPGA Mezzanine Card (FMC) connector only a simple chip carrier PCB is required. A 1 and a 10 Gigabit Ethernet interface handle the connection to the back-end. These can be used simultaneously for high-speed data and configuration over separate channels. In addition to the FMC connector, configurable inputs and outputs are available for synchronization with other detectors. A high resolution (≈ 27 ps bin size) Time to Digital converter is provided for time stamping events in the detector. The SPIDR system is frequently used as readout for the Medipix3 and Timepix3 ASICs. Using the 10 Gigabit Ethernet interface it is possible to read out a single chip at full bandwidth or up to 12 chips at a reduced rate. Another recent application is the test-bed for the VeloPix ASIC, which is developed for the Vertex Detector of the LHCb experiment. In this case the SPIDR system processes the 20 Gbps scrambled data stream from the VeloPix and distributes it over four 10 Gigabit Ethernet links, and in addition provides the slow and fast control for the chip.


Introduction
The SPIDR readout was initially designed to read out the Timepix3 [1] ASIC and had to be able to cope with the 80 Mhit/s data rate. The system was developed in parallel with the ASIC so that it could be used as soon as the chip came back from the foundry. The characterization and testing of Timepix3 started within weeks of its arrival. Since the readout was set up in a generic way, it was chosen to support the Medipix3 [2] as well. The front-end block takes care of receiving the data-stream from the ASIC. In case of the Medipix3 the interface differs significantly from the Timepix3 so a new dedicated front-end block was added. Most of the other blocks are common to the Timepix3 firmware. This allows the software to support both versions. After the success of the Timepix3 and Medipix3 readout it was decided to use the SPIDR readout as a platform for other pixel detectors. The SPIDR readout includes features like the 1 gigabit Ethernet (1 GbE) and 10 gigabit Ethernet (10 GbE), flash and DDR3 memory which are often needed in detector readout systems. The use of Ethernet means standard hardware can be used in the data acquisition system. A PC with a 1 or 10 Gb Ethernet connection is sufficient. The Ethernet interface also allows the use of optical links and hence the detector can be placed far away from the DAQ which is convenient in, for example, test beam experiments.

Background on pixel ASICs
The pixel ASIC represents a wide range of detectors from CMOS imaging or monolithic sensors to hybrid assemblies with a separate sensor and readout ASIC. In the field of particle physics, the hybrid option is often used and the examples here like the Timepix/Medipix and VeloPix ASICs are of this type. The Timepix3 and Medipix3 ASICs have distinct modes of operation. The Medipix is a "frame based" readout (like an imaging sensor) and the Timepix3 and VeloPix use a "data driven" architecture. This means, that after a pixel is hit it will send a packet containing the pixel coordinates and other hit information. The other hit information can be for example the TOT (Time Over Threshold), this will give information about the energy, and TOA (Time Of Arrival), this tells when the particle has arrived. To give an example: the Medipix3 collects counts during the shutter period and transmits the entire frame consisting of 256 * 256 * 12 bit. The bandwidth of the Medipix is ≈ 1 Gb/s (128 Mb/s * 8 links) so the frame-rate can go up to 1300 frames/s. The Timepix3 can generate 80 Mhit/s and the packet from the pixel is 48 bit. This packet contains both the TOT and the TOA. The TOA information is enough to bridge the transit time in the ASIC, but for further transmission the time-stamp information has to be extended. This happens in the FPGA and adds an additional 16 bit per pixel packet. The expected data rate on the Ethernet interface is (80 Mhit/s * 64 bit) 5.12 Gb/s for a single chip.

Hardware
The SPIDR system is based on the Xilinx 7-series FPGAs. This allows the choice of a wide range of FPGAs that can be used in the readout system with minimal firmware changes. Currently, the SPIDR firmware is built for the Virtex 7 (XC7VX485T) FPGA on the Xilinx VC707 development board and the Artix 7 (XC7A200T) FPGA on the Compact-SPIDR board.

Xilinx VC707 development board
The VC707 development board is an "off the shelf" component. With the SPIDR firmware it is possible to read out 2 Timepix3 chips at full speed (2 × 80 Mhit/s) using both FMC connectors. This system is for instance used for the wafer testing of the Timepix3 ASIC, and the Timepix3 based beam-telescope of LHCb. The dedicated transceivers on the Virtex 7 can operate at a data rate of up to 12.5 Gb/s.

Compact-SPIDR
The Compact-SPIDR PCB (figure 1) was designed at Nikhef to fulfil the need for a more portable system. The VC707 board has inconvenient mechanical properties and can be hard to mount in a detector. The Compact-SPIDR PCB has the same width as the chip-boards and in the case of the Timepix3 and Medipix3 quad chip-boards, it can go either behind or below the detector thanks to the flex-rigid design. The Compact-SPIDR board is still based on the FMC connector standard and a Xilinx 7-series FPGA, giving a high level of compatibility with the VC707 board. Apart from the smaller size, the Compact-SPIDR board has dedicated functionality for reading out a pixel chip, like dedicated power lines and sensors as illustrated in figure 2. The dedicated power for the ASIC is provided by a LTM8033 Ultralow Noise buck converter on a mezzanine board and can be adjusted  (by exchanging a resistor) from 0.8 V to 8 V and deliver up to 3 A. The Artix 7 FGPA used on the Compact-SPIDR incorporates dedicated transceivers capable of data rates up to 6.6 Gb/s.

Chip boards
For the various project different chip boards (figure 3) were produced. These range from "simple" single chip boards to advanced "flex-rigid" designs like the Timepix3 quad board. The Timepix3 quad board provides 12 high speed (640 Mb/s) links and has flexibility in the assignment of the number of links per chip.

SPIDR firmware building blocks
The SPIDR firmware consists of a combination of several building blocks as shown in figure 4. These blocks can be VHDL or VERILOG code performing different tasks. For new designs the generic blocks can be reused and only the application/ASIC specific blocks have to be added. For example, in the case of the Medipix3 the fast control/trigger block accepts external/internal triggers and generates the required shutter signal. Another key building block is the LEON3 open-source 32-bit synthesizable processor core based on the SPARC V8 architecture developed by Cobham Gaisler [6] (figure 4). The core is highly configurable and only uses a small percentage of the FPGA resources (2% of LUTs and 7% of RAM in the Virtex 7 XC7VX485T). Another important block is the UDP offload block. Here the received data from the ASIC is packed after processing in UDP frames. There is a time-out to transmit the packet when there is no -4 -new data for a set time. An Ethernet packet has a header containing several fields (addresses, CRC checksums). In order to maximize the available data bandwidth one should maximize the payload size. The size of the payload can be configured up to the full "Jumbo frame" (9000 bytes) size. The bigger payload versus header means less overhead and thus increases the available bandwidth. Using UDP packets the bandwidth can reach almost (99%) of the 1 GbE or 10 GbE speeds.

The API
For the SPIDR system a C++ "Application Programming Interface" (API) was developed to control the SPIDR system on the PC side. The QT cross-platform class library [9] is used to provide operating-system independent functionality. Using this API, commands can be transmitted and received using a TCP/IP connection over the 1 GbE or 10 GbE link. Using TCP/IP assures the lossless and reliable transmission of packets which is required for the configuration data. The high speed data links use the UDP protocol. Since there is much less overhead and acknowledgements are not required in this protocol more bandwidth is available. The drawback is a potential loss of packets, which can not be recovered. Monitoring counters are implemented to signal the loss of packets. Experience has shown that packet loss is virtually zero for a point to point connection.
The API also provides the Data Acquisition (DAQ) classes to receive the high speed stream of UDP packets. The API controls both the SPIDR board and the ASICs. A large part of the API can therefore be reused for other ASICs. Using the API, "simple" programs can be written like a DAC panel or threshold equalization routine. The LEON3 CPU can handle multiple connections over TCP/IP simultaneously. Integration in larger DAQ or experiment control systems is also possible via this API.

The LEON3 CPU
In the SPIDR system the LEON3 "soft core" CPU is used. The LEON project was originally started by the European Space Agency and was later continued by Cobham Gaisler. The LEON3 is released under Open-source licensing for research and education. Fault tolerant implementations of the LEON3 also exist for space applications and it can be implemented both in FPGAs and ASICs. The high degree of flexibility and the availability of the source-code makes it convenient and "FPGA independent" compared to for example the Xilinx Microblaze.
The CPU software implements the TCP/IP stack and can provide both the ASIC slow control and periphery device configuration like I 2 C and SPI commands. The LEON3 runs a C++ program compiled with the Bare-C Cross-Compiler System provided by Cobham Gaisler. The tool-chain uses the Java based Eclipse IDE and is operating system independent. The LEON3 is capable of running an OS, like Linux or VxWorks but the memory resources available to the LEON3 in SPIDR are not sufficient. The interface to the FPGA logic is a 32 bit memory mapped set of registers. To transmit data to the ASIC a FIFO is filled with the payload and the transmit state-machine is started when done. The FIFO is used to move the data across clock domains. This ensures the transmitted data is free of gaps and is clocked to the proper clock domain. Receiving data from the ASIC happens in a similar way using a FIFO.

Timepix3 and Medipix3 compact readout system
The Compact-SPIDR is in use by multiple institutes and commercial companies for various experiments and R&D activities. The Compact-SPIDR system is available to members of the Medipix collaboration [4] and is available for commercial parties through Amsterdam Scientific Instruments (ASI) [7]. The performance specifications are: • 1X Timepix3 at 80 Mhit/s "data-driven" 1.6 ns TOT+TOA • 4X Timepix3 at 32 Mhit/s "data-driven" 1.6 ns TOT+TOA • 4X Medipix3 at 1.3 kframes/s sequential and continuous mode.

LHCb Beam-telescope
The LHCb Beam-telescope is used for the sensor characterization campaign in preparation for the Vertex Detector [10] upgrade project. It consists of 8 Timepix3 planes plus a device-under-test (DUT). There are 4 VC707 SPIDR systems to read out the four planes in each arm and a 5th for the DUT. It has been operational for 2 years at a permanent location at SPS beam line H8 [11]. The telescope is able to process 10 million tracks/s in data-driven readout mode.

VeloPix test system
In parallel with the VeloPix [8] chip design for the upgrade of the LHCb Vertex Detector the SPIDR adaptation was started. This meant there was a functional readout system as soon as the ASIC was delivered. In order to cope with the maximum data rate of 20 Gb/s, four 10 GbE links are used. In this case, the Front-end and Slow-control blocks were rewritten for this ASIC. There are four scrambled data streams at 5.12 Gb/s coming from the VeloPix. In the front-end block this data stream is de-serialized using the transceiver core of the Virtex 7 FPGA. After de-serialization, data is word-synchronized and descrambled. The descrambled data gets some extra time-stamp information and is packed in Ethernet frames. Because there is no need for a path to the Slowcontrol/CPU, less flexibility is needed and fewer FPGA resources are used than the Timepix3 implementation. The Slow-control block had to be rewritten for this specific (ECS) interface but it works similarly to the Timepix3 block. The system has been in use to perform the testing and characterization of the VeloPix [3] and will be used for the wafer probing as well.

Conclusions and recommendations
In conclusion, the SPIDR system provides an excellent base for the readout development for new and existing detector ASICs. Many design blocks and functionality can be re-used. By complying to existing standards like FMC and Ethernet, effort in designing interfaces is minimized. The same argument holds for the design of the software interface. Having a transparent way of communicating with the ASIC from the software is convenient for the ASIC testing. This approach has been applied successfully in several projects.
-6 -To use SPIDR as a readout platform for new ASICs, "early involvement" of the readout designer is important given the complexity of the ASICs. Starting the readout development when the chip is taped out means the readout will be finished too late to start testing immediately. Getting to know the ASIC from simulations even before the design is finished gives more time to design the readout architecture. The early involvement also means there is an extra verification step in the chip design. Mistakes or other "features" can be discovered during the design of the readout system.
A second factor in readout design is the re-use and smart implementation of design blocks. Using high level coding the firmware can be easily adapted for use in different applications. With high level coding it is meant the SPIDR system can use a configurable amount of data-links/number of chips or use different link speeds. The signals controlling these statements can be set during compile time so there is no need to change the source code for this kind of changes.