The FC7 AMC for generic DAQ & control applications in CMS

The FC7 is a flexible, μTCA compatible Advanced Mezzanine Card (AMC) for generic data acquisition/control applications. Built around the Xilinx Kintex-7 FPGA, the FC7 provides developers with a platform which has access to a large array of configurable I/O, primarily delivered from on-board FPGA Mezzanine Card (FMC) sockets. Targeting users of high-speed optical links in high energy physics experiments, the board is capable of driving and receiving links up to 10 Gbps. This paper presents test results from the first set of pre-production prototypes and reports on FC7 uses and applications towards upgrades in CMS.

. Picture of the FC7 R0 board.
• Xilinx Kintex-7 KC7K420T-2 FPGA -capable of supporting line rates of up to 10 Gbps • card edge AMC [4] connector -provides high-speed connectivity on up to 12 ports and is fully compatible with the AMC13, µTCA for CMS applications, and supports a range of MGT protocols to the backplane • two 400-pin Samtec SEARAY sockets -chosen to be fully compatible with Low Pin Count [3] (LPC) FMCs but to additionally provide access to all high-speed serial lines as defined by the High Pin Count [3] FMC pinout, so that a maximum of 20 lines are available on the front panel. Accommodate dual width, legacy and non-standard FMCs • 4 Gb DDR3 RAM -capable of memory transfer rates of up to 30 Gbps • microSD slot -provides storage for a firmware image repository including an inviolate fallback 'golden'image, and reduces firmware write times down from 5 minutes to 10 seconds • ATMEL microcontroller -implements Module Management Controller (MMC) functionality, as defined by the AMC specification. Runs common software developed with the MP7.

Manufacturing experience & acceptance testing
A total of 4 prototype boards and 15 pre-production boards have been manufactured with two different suppliers. This is both to spread risk and to help distinguish between manufacturing and design faults (whether due to designer error or board complexity). Two alternative PCB materials have also been trialled in order to maximise the possibility for good signal integrity at high frequencies, impedances that meet requirements, high manufacturing yield and fast turnaround. Nelco N4000-EPSI has been the material of choice for Xilinx Series-7 development cards up until recently while TUC TU-872 SLK offers similar high frequency performance with better manufacturability. Some of the earlier prototypes manufactured using Nelco N4000 laminate exhibited delamination during assembly (figure 2). According to the material manufacturer, risk of delamination is increased in high layer count PCBs with buried or blind vias (such as the FC7) when using -2 -  Nelco N4000 laminates. This can impact on the final yield and cost of the project, especially if the FPGA has already been mounted. After defining careful handling procedures for manufacturing and assembly, including guidelines for baking, humidity control and temperature profiles, risk of delamination has been reduced and no effects were reported during assembly of the 10 pre-production boards using Nelco. Five pre-production boards were also manufactured using TUC TU-872 laminate. No issues with delamination were reported during assembly although registration defects with this material have been reported since. A larger production using this laminate would be required in order to confidently qualify the yield, however the pre-production batch has been sufficient for quantifying the performance of the PCB, especially with respect to high frequency signal integrity (section 4.3).
A plug-in automated acceptance test stand has been assembled at CERN for qualification of the production boards ( figure 3). The test stand includes power-on current and voltage monitoring via a GPIB-controllable power supply, automatic programming of all ICs including the FPGA, full configuration and control of the board over a Gigabit Ethernet link, and board self-tests through the use of external loopback cards. The test takes approximately 5 minutes and qualifies almost all of the board functionality.

High speed testing
The FC7 provides up to 20 high-speed serial links to the front panel and up to 12 high-speed serial links to the backplane. 1 The front panel and backplane links have been specified to run at line rates of up to 10 Gbps and 5 Gbps respectively, and so this section describes the high-speed performance of the board under electrical loopback, as well as for a couple of typical use cases. To emulate 64b66b encoding schemes, Pseudo Random Bit Sequences of length 2 31 -1 bits (PRBS-31) are used for all measurements of bit error rates using the internal Kintex-7 margin analysis tools. When measuring the electrical eye and quantifying the jitter using the oscilloscope, PBRS-7 patterns must be used due to the limitations of the device. 2

Front panel performance
To illustrate the electrical performance of the FC7 at 10 Gbps, the Xilinx XM104 FMC is used to provide access to two high-speed serial lines (DP0, DP1). Using a PRBS-7 pattern sequence, the electrical eye has been captured on SMA cables for these channels using a DSA91204A high performance oscilloscope on both the L8 (bottom) and L12 (top) FMC sockets. The electrical eye diagrams from all four transmit channels demonstrate excellent high frequency characteristics, minimal attenuation and low jitter. The average total jitter across all channels is 20.0 ps with a channel to channel spread of 1.7 ps (figure 4).
Using the same FMC, the signals can be looped back to demonstrate the ultimate performance of the board under electrical loopback via an FMC. A new feature for the Kintex-7 GTX transceivers is the ability to perform margin analysis via internal scanning and statistical rebuilding of the electrical eye at the receiver using Bit Error Rate (BER) measurements. Under electrical 1 Although for CMS applications Port 3 is configured as a low-speed LVDS link. 2 The run length of PRBS-31 patterns is usually too large for oscilloscopes to acquire sufficient data to perform Data Dependent jitter analysis.
Reference SFP FM-S18 Figure 5. Left: setup for measuring the performance of the FC7 with FM-S18 FMC as optical receiver at 10 Gbps. The Kintex-7 KC705 evaluation board is used as reference transmitter. Right: setup for measuring the relative performance of the KC705 with FM-S18 FMC as optical receiver at 10 Gbps using the same reference transmitter. loopback, using a PRBS-31 data pattern we estimate the total jitter to be 0.36 Unit Interval (UI), or 36 ps. The eye opening is therefore 0.64 UI, providing a benchmark for the ultimate margin of the FC7 when operating the transceivers at 10 Gbps across the FMC interface.
If operation at 10 Gbps is required, the end user will typically choose an optical transmission interface. Since there are a wide range of optical transceivers available on the market and numerous communication standards and protocols to choose from, for the purpose of demonstrating the typical operating performance of the FC7, a commercial FMC using the common SFP+ standard is used. The Faster Technology FM-S18 FMC provides access to 8 channels on each FMC socket, while two Avago 850nm AFBR-703ASDZ SFP+ modules are used.
The Xilinx Kintex-7 KC705 evaluation board is used to provide both a reference measurement with the FM-S18 FMC (section 4.3), and to act as the reference transmitter via the on-board SFP in order to decouple possible effects due to the FMC ( figure 5). For all tests, one SFP+ is always used as the transmitter and one always as the receiver, while the optical medium is 10 m OM2 multimode fibre. For consistency, the same 250 MHz MGT reference clock (on-board the FM-S18) is used by the FPGA to receive data on the FMC channel under test. 3 The Kintex-7 transceivers provide a range of settings in order to fine tune each channel to extract the widest possible operating margin. For the purpose of these tests, minimal settings are applied to replicate an out-of-the-box use case. The transmitter differential swing is set to 600 mV (recommended AFBR-703ASDZ range is 180-700 mV) and no pre-or post-de-emphasis is applied. All channel receivers are set to operate in Low Power Mode (no decision feedback equalisation).
Using PRBS-31 data patterns, eye openings for all channels (L8 and L12) on the FC7 at the receiver are compared in figure 6, and plotted as a function of trace length. Using the FM-S18 FMC, margin varies between 0.48 UI and 0.34 UI and tends to be lower for the longest trace lengths. The channel with the smallest margin has been tested in loopback. No single bit error was seen in 10 14 transmitted bits.
Replacing the 10 m fibre with an optical attenuator, it is possible to measure the receiver sensitivity of the FC7 with the FM-S18 FMC for the channel with the smallest margin. Figure 6 de-   scribes the BER measured by the FPGA, as a function of the Optical Modulation Amplitude (OMA) seen by the SFP+ receiver for a transmitted PRBS-31 data pattern. Given the stated worst case sensitivity of the SFP+ of -11.1 dBm, the results show that the FC7 with FMC does not significantly degrade the performance of the optical transceiver, requiring a minimum optical input power of approximately -13 dBm for a 10 −12 bit error rate. A system built on these components would provide a large optical margin of up to 10 dB (the SFP+ transmitter has an approximate -2.5 dBm OMA).

Backplane performance
The most critical channels for CMS are Ports 0 and 1, along which system communication with the MicroTCA Carrier Hub (MCH) and CMS-specific communication with the AMC13 is carried respectively. To illustrate the electrical performance of the FC7 backplane links, the Turnkey Express AMC SMA Ultra-9000 breakout board allows capture of the electrical eye. Diagrams for Port 0 at 5 Gbps and at 10 Gbps are shown in figure 7 (PRBS-7). Total jitter from the breakout board was measured to be 47.1 ps and 44.0 ps for Ports 0 and 1 at 5 Gbps (∼0.23 UI). At 10 Gbps the total jitter is measured at 37.3 ps and 35.9 ps for Ports 0 and 1 respectively (∼0.36 UI) indicating a high degree of margin, given that the links are operating above the FC7 specification (5 Gbps on the AMC connector) and the Ultra-9000 is not optimised for high frequency applications.
Typical performance of the FC7 in a CMS application has been measured using a Vadatech VT892 µTCA crate with an AMC13XG occupying MCH slot 2 and the FC7 occupying the furthest possible slot (figure 8). Using a PRBS-31 data pattern, error-free transmission of over 10 13 bits was observed from the FC7 to AMC13XG (and vice versa) on Port 1 at 5 Gbps. Figure 8 indicates that the eye opening across the backplane is at least 0.45 UI wide.

Initial comparison of PCB materials
To compare the high frequency performance of the two FC7 PCB builds under a typical end-user scenario, 'bath-tub'scans, using the FM-S18 FMC, are performed replicating the full 2D eye scan in the time axis only. The measurement uses PRBS-7 pattern sequences, which is sufficient for a general comparison of this nature. Figure 9 shows the result of the scan for the KC705 reference channel under optical loopback, alongside the four (out of eight) channels accessible to the KC705 using the FM-S18, when using the reference channel as transmitter. It can be observed that there is an approximate 25% degradation in the eye opening down to 0.44 UI when using the FM-S18 FMC compared to using the on-board SFP. Figure 9 also demonstrates the scans for the four comparable channels on the KC705 and the FC7 for both Nelco and TUC builds (PRBS-7). For clarity only results from the channels on the L8 socket are shown. Qualitatively it can be seen that as a carrier, the performance of the FC7 in this configuration is at least as good as the Xilinx evaluation board in an equivalent configuration. 4 Additionally, there does not seem to be any significant difference between the two FC7 laminates tested, indicating that manufacturing using either material would not adversely impact upon performance under typical use cases. . Left: bath-tub scans, using the Xilinx internal margin analysis tools, for the KC705 with FM-S18 FMC at 10 Gbps (PRBS-7, channels DP0-3), compared to the KC705 reference SFP operating in loopback. There is an approximate 25% degradation in the eye opening when using the FMC, compared to the onboard SFP. Right: bath-tub scans comparing the performance of the FM-S18 at 10 Gbps (PRBS-7, channels DP0-3) on the KC705, FC7 (Nelco build) and FC7 (TUC build).

System infrastructure
In order to simplify the application-specific development process for users, the FC7 follows a firmware architecture (inherited from the GLIB) whereby the FPGA design is divided into two blocks; the system core and the user logic (figure 10). The system core, delivered by the FC7 team, implements all basic infrastructure such as the Gigabit UDP Ethernet link based on IPBus (a simple IP-based control protocol designed for controlling xTCA-based hardware) [8], the configuration of the clocking circuitry, the control of the powering circuitry, the MMC communication, the SD card programming, board identification and various monitoring tasks. Additionally, the FC7 firmware provides modules that can be implemented in the user logic if needed, including the interface with the DDR3 memory and generic I2C and SPI controllers.

Example use in CMS
The FC7 is currently being deployed as part of the Trigger Control and Distribution System [6] (TCDS) upgrade for CMS and is due to be fully installed and commissioned in time for data taking in 2015. In the TCDS, the FC7 satisfies the role of two of the three objects in the system via the use of custom made FMCs and specialised firmware.
The flexibility afforded by the use of FMC mezzanines makes the FC7 well suited to the support of both legacy and future CMS hardware in an experiment-wide common system such as the TCDS. The specific challenges for this project are ensuring deterministic trigger latency, clock phasing and frequency, minimising link jitter and minimising disruption to the existing hardware. For this purpose a demonstrator system (figure 11) is currently installed in the Service Cavern at Point 5 for integration with the rest of CMS.  Figure 10. High level overview of the firmware architecture available for use with the FC7. The FC7 is a flexible FPGA-based µTCA compatible AMC targeting generic data acquisition and system control uses in CMS or other experiments. The board has been produced on a short timescale -development began in late 2012. A successful pre-production run of 15 boards trialling two different PCB materials and alternative suppliers in early 2014 has allowed a full qualification of the board performance and functionality while helping to identify potential manufacturing issues early on. A fully automated 'plug-in'acceptance test stand has been assembled at CERN and developed using the pre-production boards.
Extended testing indicates the FC7 is suitable for applications requiring high-speed optical links running at line rates of up to 10 Gbps. Performance tests indicate wide margins of operation on all channels and error free running with up to 10 14 bits transmitted per link.
A production run of 70 boards with one supplier using Nelco N4000 PCB laminate was completed in June 2014. Including the 10 pre-production boards, total yield with this supplier was above 97%. These boards are currently being installed in the USC at P5 as part of the TCDS upgrade for CMS. A second production of 120 pieces is underway and pre-production boards are due to arrive in December 2014. These boards will be used as part of the CMS Pixel detector DAQ upgrade, due for installation in 2016-2017.