The Giga Bit Transceiver based Expandable Front-End (GEFE)—a new radiation tolerant acquisition system for beam instrumentation

The Giga Bit Transceiver based Expandable Front-End (GEFE) is a multi-purpose FPGA-based radiation tolerant card. It is foreseen to be the new standard FMC carrier for digital front-end applications in the CERN BE-BI group. Its intended use ranges from fast data acquisition systems to slow control installed close to the beamlines, in a radioactive environment exposed to total ionizing doses of up to 750 Gy. This paper introduces the architecture of the GEFE, its features as well as examples of its application in different setups.


Introduction
The Giga Bit Transceiver based Expandable Front-End (GEFE) (see figure 1) is a multi-purpose FPGA-based radiation tolerant card, available under the CERN Open Hardware License (CERN OHL) [1]. It is foreseen to be the new standard FMC carrier for digital front-end applications in the CERN BE-BI group (equivalent to what the VFC-HD [2] is for the back-end applications). Its intended use ranges from fast data acquisition systems to slow control installed close to the beamlines, in a radioactive environment exposed to Total Ionizing Doses (TID) of up to 750 Gy. Figure 2 illustrates the diagram of a typical system for beams instrumentation using the VFC-HD and the GEFE for back-end/front-end communication. The GEFE takes advantage of the Giga Bit Transceiver (GBT)/Versatile Link developed by the CERN PH-ESE group [3]. This provides a rad-hard, high-speed (4.8 Gbps), bidirectional optical link (named Versatile Link) for back-end/front-end communications. The communication through the GBT/Versatile Link can be fixed and deterministic in clock phase and data latency if required. In addition to the GBT/Versatile Link, the GEFE features a custom Electrical Serial Link Transceiver (ESLT), to be used in low-speed communications over copper cable through long distances (tested up to 2 km at 10 Mbps). In order to maximize the versatility, the GEFE is expanded with dedicated mezzanine cards through a High-Pin Count FPGA Mezzanine Card (FMC HPC) connector [4]. This feature gives -1 -2016 JINST 11 C02062 users the possibility of having an application specific digital or mixed-signal system, for interfacing with front-end electronics.
The use of an FPGA, coupled with flexible powering, clocking and FPGA programming schemes, provides the capability to adapt the GEFE for interfacing to the user's systems. Moreover, the variety of optical and electrical interfaces on the board, in addition to its flexible architecture, mean that it can easily be adapted for use in many different applications where radiation tolerance is a requirement.

Implementation
The GEFE presents a small custom form factor (200 × 100 mm) conceived to operate inside a small "pizza box" in the front-end, close to the beamlines. Additionally, these dimensions allow the GEFE to be inserted in an Eurocard 3U [5] compatible crate if required. Figure 3 shows a block diagram of the GEFE, highlighting its mayor components.

FPGA-based board
The board is based on the ACLA3PE3000-FGG896 from the ProAsic3 family by Microsemi [6], a flash-based FPGA that has been qualified for operating in radioactive environments. The main function of the ProAsic3 FPGA in GEFE is to interface the GBT/Versatile Link (or the ESLT) with the digital front-end electronics of a mezzanine card plugged onto the FMC HPC connector. In addition, the different on-board resources of the GEFE are also connected to the ProAsic3 FPGA (e.g. DIP switch, GPIO connectors, etc.). The configuration of the ProAsic3 FPGA may be accomplished by either a Microsemi JTAG programmer via an on-board 10-pin connector or a JTAG master on a hosted FMC.

The Giga Bit Transceiver (GBT)/Versatile Link
The GBTx [7], one of the two main components of the GBT/Versatile Link, is a radiation tolerant chip that can be used to implement multi-purpose, high speed (4.8 Gbps), bidirectional, optical links -3 -for high-energy physics experiments and beamlines. On one side, the GBTx communicates with an FPGA-based back-end board (e.g. VFC-HD), configured with the GBT-FPGA core [8]. This communication is done through a high-speed optical link (the Versatile Link), using a radiation tolerant optical transceiver named VTRx [9], the other main component of the GBT/Versatile Link. If required, the GEFE may be set to host a standard Small Form factor Pluggable Plus (SFP+) [10] optical transceiver to replace the VTRx. On the other side, the GBTx communicates with the front-end electronics (in this case with the ProAsic3 FPGA) through multiple low-speed electrical links named e-links. In GEFE, all user e-links are connected to the ProAsic3 FPGA, whilst the Slow Control (SC) e-link is forwarded either to the ProAsic3 FPGA (default) or to the FMC HPC connector. This configuration brings the possibility of having an FMC featuring the GigaBit Transceiver -Slow Control Adapter (GBT-SCA) ASIC [11] plugged onto the GEFE. It is important to mention that a GBT-SCA mounted on an FMC may be used for remotely reconfiguring the ProAsic3 FPGA of the GEFE, using the programming JTAG chain. The configuration of the GBTx is done by I2C protocol though either a dedicated 8-pin connector using the USB-I2C CERN dongle [3] or a direct connection with the ProAsic3 FPGA.

The Electrical Serial Link Transceiver (ESLT)
Besides the GBT/Versatile Link, the GEFE board features the ESLT, enabling serial data transmission/reception at low-speed over copper cable through long distances (tested up to 2 km at 10 Mbps). Originally, this fully custom electrical link based on discrete components was developed at CERN by the BE-BI-QP section, with the purpose of communicating the back-end with the front-end cards in the BPM system of the CERN Neutrinos to Gran Sasso (CNGS) project [12]. Currently, the ESLT is used in the BPM system of the Advanced Wakefield Experiment (AWAKE) [13]. This project reuses the CNGS facility, including the installed copper cables for the back-end/front-end communication.

The High Pin Count FPGA Mezzanine Card (FMC HPC) connector
For interfacing with other front-end electronics, the GEFE makes use of the FMC HPC connector, defined in the ANSI/VITA 57.1 standard. The FMC HPC connector features up to 160 user-specific I/Os (that can be configured both as single-ended or differential pairs) as well as 4 differential clock inputs and 2 differential bidirectional clocks. In GEFE, the high-speed lines (DP) of the FMC HPC connector do not comply with the ANSI/VITA 57.1, since they are used as user-specific I/Os and can also be used for special functions (e.g. SC e-link to FMC HPC, ProAsic3 FPGA programming from JTAG master on FMC, etc.).

Other general purpose connectors
In addition to the FMC HPC connector, the GEFE offers 2 general purpose connectors. Both presenting the same form-factor (26-pin with a pitch of .050 ), though one of them features 13 noise shielded user-specific I/Os (where 8 of these I/Os may be directly connected to the FMC HPC), whilst the other connector features 24 user-specific I/Os.

Clocking scheme
Due to its general purpose nature, the GEFE presents a very flexible clocking scheme where the GBTx may be used as main clock source by the ProAsic3 FPGA and thus the electronics modules plugged onto the different on-board connectors. The GBTx generates the "e-link clocks" (DCLK) and the "slow control e-link clock" (SCCLK), with frequencies set accordingly to the line rate of the e-links and the slow control e-link respectively. Moreover, the GBTx also generates the "programmable clocks" (CLOCKDES), with frequencies and phases configurable by the user. The GBTx itself may receive its reference clock from three different sources: an on-chip crystal oscillator running at the LHC frequency (40.08 MHz), an LVDS clock from an external source through two Micro-Miniature CoaXial (MMCX) connectors [14] or the recovered clock from the high-speed optical link (Versatile Link) through the on-chip Clock and Data Recovery (CDR) circuitry. On the other side of the GEFE, all dedicated clocks and clock capable user I/O pins of the FMC HPC connector are also connected to the ProAsic3 FPGA, enhancing the versatility of the GEFE when interfacing the front-end FMCs. As special function, for cases where very low jitter (in the order of ps) is a critical factor and the forwarded clocks from the ProAsic3 FPGA do not comply with the requirements, a dedicated CLOCKDES may be directly forwarded from the GBTx to the FMC HPC connector through one of the DP lines. Besides the GBTx and the FMC HPC connector, the ProAsic3 FPGA receives other auxiliary reference clocks from the following sources: an onboard crystal oscillator running at 25 MHz, an LVDS clock from an external source through two MMCX connectors and four single-ended clocks from external sources through respective MMCX connectors.

Resets scheme
For simplicity reasons, the reset scheme of the GEFE may be divided into two different parts. On one side, the reset scheme of the ProAsic3 FPGA and on the other side the reset scheme of the other components (e.g. GBTx, FMC HPC connector, etc.). For the ProAsic3 FPGA, the power-on/post-programming (POR/PPR) reset follows the scheme proposed in the application note AC380 from Microsemi [15]. Additionally, the Loss Of Signal of the VTRx (RSSI) or the SFP+ (LOS), that indicates the level of the optical signal in the receiver, may be used as asynchronous reset by the ProAsic3 FPGA. This feature allows the remote initialisation of the FPGA logic from the back-end, by stopping the optical data transmission of the downstream or unplugging the optic fibre. Both reset signals, the POR/PPR and the RSSI/LOS (as well as any other reset source) may be forwarded out from the FPGA thowards the GBTx as well as the FMC HPC and/or the general purpose connectors, in order to synchronize the initialization of the different components. Besides the previously mentioned reset from the ProAsic3 FPGA, the GBTx is reset at power up by the power good signal of its power source and may also be reset by an on-board push button for test purposes.

Powering scheme
In addition to its flexible FPGA programming, clocking, and reset schemes, the GEFE also presents the same philosophy on the powering scheme. The main power of the board, provided through a dedicated connector, may range from 5 to 12 volts. It is used for powering two radiation tolerant -5 -DCDC converters from the FEASTMP family [16], developed at CERN by the PH-ESE group and a radiation tolerant Low Drop Output (LDO) voltage linear regulator named LHC4913 from STMicroelectronics [17], widely used at CERN. One of the FEASTMP (set to 1.5 V) is used for powering the GBTx and the core of the ProAsic3 FPGA. Optionally, this DCDC converter may power the Vadj pins of the FMC HPC connector and its corresponding I/O banks of the ProAsic3 FPGA. The other FEASTMP (set to 2.5 V) is the power source of the optical transceiver when using the VTRx as well as it is also the default power source for the Vadj pins of the FMC HPC connector and its corresponding I/O banks of the ProAsic3 FPGA. The LHC4913 (set to 3.3 V) is used for powering the 3.3 V and 3.3 V Aux pins of the FMC HPC connector as well as the Vadj pins and its corresponding I/O banks of the ProAsic3 FPGA if required. Furthermore, this linear regulator may be used for powering the optical transceiver when using an SFP+ instead of the VTRx. Besides sourcing the two FEASTP and the LHC4913, the main power of the GEFE is also forwarded to the 12 V line of the FMC HPC connector (Please note that providing any other voltage level than 12 V is not compliant with the ANSI/VITA 57.1 standard). In cases where the FMC requires 12 V and the main power has a different voltage, the 12 V power line of the FMC HPC may be decoupled from the main power and connected to a dedicated 12 V source though a secondary on-board connector.

Other features
Finally, as any other general purpose board, the GEFE also features the typical I/O resources, such as push button, DIP switch or LEDs.
Further information about GEFE's features and architecture (such as the special functions of the FMC DP lanes) can be found in the GEFE specifications [18].

Radiation tolerance
One of the techniques used on the GEFE board in order to maximize the radiation tolerance is to reduce the number of active components as much as possible in terms of quantity and type.
These active components may be divided in two groups. On one hand, the rad-hard by design qualified in terms of radiation tolerance (e.g. TID, displacement damage or Single Event Effects (SEE)) and presenting mitigation techniques (e.g. Triple Module Redundancy (TMR), error detection/correction encodings, etc.) for minimizing SEE cross section when required. On the other hand, the Components-Of-The-Shelf (COTS) with neither qualification in terms of radiation tolerance nor mitigation techniques by design. For that reason, the COST selected for the GEFE board are tested in radiation in order to guarantee the target levels of radiation tolerance.
In GEFE, the critical COST (all CMOS-based) are not affected by displacement damage and present very low SEE cross section [21]. Moreover, the ProAsic3 FPGA (the main COST of the GEFE board) features flash-based configuration memory immune to SEE as well as mitigation techniques may be implemented in the user logic [25]. Please note that the maximum levels allowed of SEE cross section are application dependent.
The TID is the most critical radiation effect, since it determines whether the COST will withstand the expected doses of radiation without permanent damage. The target TID level of the GEFE board has been set to 750 Gy. This level is a limitation imposed by the ProAsic3 FPGA. The presence of an FPGA reduces the tolerance in terms radiation but increases the flexibility of -6 - Not qualified yet the board. It allows an easy implementation of the glue logic required to interface the rest of the front-end system with the GBTx. The qualification of the COTS that have not been already tested up to the target TID levels is being carried out at the Paul Scherrer Institut (PSI) [19]. There, in one of the beamlines of the COMET [20], a cyclotron that delivers a proton beam of an energy of 250 MeV, the different COTS of the GEFE are irradiated until reaching the target TID of 750 Gy. Please note that all COTS selected for GEFE except one (SN65LVDS2DBVR) were already qualified by other CERN teams or external institutes, but for radiation levels below GEFE's target TID. At this stage of the project, almost all COTS used in GEFE have been sucesfully qualified by the GEFE team, with the collaboration of the CERN's Radiation Working Group (RadWG) [21], passing the radiation test with no degradation of the different parameters. Only two COTS have not been qualified up to GEFE's targer TID yet, a NMOS transistor (to be selected) and a LVDS receiver (SN65LVDS2DBVR). These components are foreseen to be tested by beginning of 2016. Table 1 enumerates all active components of the GEFE, indicating their maximum radiation tolerance in terms of TID.

Application examples
As previously mentioned, the GEFE is foreseen to be the new standard FMC carrier for digital front-end applications in the CERN BE-BI group. Thanks to its variety of optical and electrical interfaces, coupled to a flexible architecture, the GEFE can easily be adapted for use in many different applications where radiation tolerance is a requirement.
So far, two BE-BI projects (New Multi-Orbit POsition System SPS (New MOPOS SPS) [22] and Motor Controller with Optical Interface (MCOI) [23]) have adopted the GEFE board, whilst seven other projects from BE-BI and other groups at CERN (Cern High energy AcceleRator Mixed field facility (CHARM) [24] test board, AWAKE BPM, etc.) are evaluating the use of GEFE. Figure 4 shows the block diagrams of four possible applications of the GEFE board. Example A depicts the New MOPOS SPS system, where the GEFE communicates with the back-end systems though the GBT/Versatile Link. On the other side of the board, a hosted FMC featuring ADCs samples the analogue signals (previously conditioned by the front-end analogue electronics) coming from a BPM. Example B shows the MCOI system, also relying on the GBT/Versatile Link for the back-end/front-end communication. However, in this case, the GEFE is inserted into an Eurocard 3U compatible crate and plugged onto a backplane through a connector placed on the FMC. This  backplane features the drivers required for controlling the electrical motors. Example C represents the AWAKE BPM system. There, the GBT/Versatile Link is replaced by the ESLT, whilst the front-end electronics connected to the GEFE board are similar to the ones used in the New MOPOS SPS system. In the last example (D), the GEFE plays the role of a devkit for the new radiation tolerant high-bandwidth field bus, a new project from CERN BE-CO, aiming to replace the current WorldFIP as front-end field bus in CERN applications [25]. With this purpose, the GBT/Versatile Link is also used for communicating the back-end front-end, whereas on the other side, multiple front-end digital cards are controlled thought their respective e-link.

Summary and status
This paper has introduced the GEFE, a multi-purpose FPGA-based radiation tolerant card available under the CERN Open Hardware License, which is foreseen to be the new standard FMC carrier for digital front-end applications in the CERN BE-BI group. Its intended use ranges from fast data acquisition systems to slow control installed close to the beamlines, in a radioactive environment exposed to total ionizing doses of up to 750 Gy. Its main features and architecture as well as some application examples have been presented. The GEFE has been adopted by two CERN BE-BI projects and is being evaluated by seven other projects at CERN. The validity tests of the two first prototypes of the GEFE v1 are being carried out. The qualification in terms of radiation tolerance is foreseen to be finished by the first half of 2016. A small pre-production batch for prototyping is scheduled by the first half of 2016, whilst the production stage will take place during the second half of 2016. We envisage that the GEFE board will be adapted to various other applications at CERN and other external institutes thanks to its radiation tolerance and variety of interfaces. More detailed information and updates can be found in the OHWR wiki [26] and mail list [27] of the GEFE project respectively.