A software package for the full GBTX lifecycle

This work presents the software environment surrounding the GBTX. The GBTX is a high speed bidirectional ASIC, implementing radiation hard optical links for high-energy physics experiments. Having more than 300 8-bit configuration registers, it poses challenges addressed by a wide variety of software components. This paper focuses on the software used for characterization as well as radiation and production testing of the GBTX. It also highlights tools made available to the designers and users, enabling them to create customized configurations. The paper shows how storing data for the full GBTX lifecycle is planned to ensure a good quality tracking of their devices.


Introduction
This paper introduces the software environment created to support all steps in the life cycle of the GBTX from characterisation and radiation testing to customization performed by the end-users. The GBTX is a high speed bidirectional ASIC, implementing radiation hard optical links for highenergy physics experiments [1]. Having more than 300 8-bit configuration registers, it poses challenges addressed by a wide variety of software components. This work discusses how those software components help the test engineers to handle the complexity of the chip, thus enabling an efficient characterisation process. Another focus rests on showing the efforts undertaken to support users in customizing their GBTX chips.
The characterisation of the GBTX relies strongly on the acquisition of data during radiation test runs. Therefore, a specific protocol has been created that allows the exchange of control signals and data packages between the testing FPGA and the software. The testing software is accompanied by several components that proved to be valuable in testing a system as complex as the GBTX. One of those elements is a software emulator that allows to mimic a subset of the chip functions. Doing that helps to simulate the reaction of the test software to conditions that are hard or impossible to reproduce on demand in the laboratory. Further powerful implementations are shown, like the development of a 'LiveViewer' providing means to analyse radiation test runs in detail almost immediately. Its functionality is based on a tailored storage system recording all data from a test run.
A website is provided to guide users in the process of creating custom-tailored configurations. It is a learning tool based on a wizard that enables users to create customized and programmable -1 - configurations step by step. A completed configuration is sent to the user in the form of a register map file that has an open self-descriptive format. Using a web-based technology represents the most flexible solution that allows users to have access to the latest software releases.
The above-described public configuration interface is connected to a central CERN database dedicated to GBTX. This storage system is as a link between all systems associated with production testing, performance data acquisition, user configuration and incident reporting of the chip. For every single device, the database contains data collected in the various stages of its life cycle. This innovative interconnection will represent a very useful solution in providing a real Quality Assurance policy to the GBTX. Having access to a knowledge database is expected to help the support team in quickly responding to user requests.

The test system
Testing the GBTX is done by using a custom-made board dedicated for this purpose as illustrated in figure 1. This so called Standalone-Tester (SAT) contains an FPGA with the GBT-FPGA [2] core implemented and a socket to hold the GBTX. Both components can be connected together by optical fibers representing the Versatile Link [3] between the GBTX and a back-end FPGA. In order to fully test the chip, the 120 differential E-Links of the GBTX are also connected to the FPGA. The E-Links represent the data exchange between the chip and the front-end systems. To configure the GBTX, an I2C bus connects the FPGA to the chip. For alternative control, the board is equipped with I 2 C connectors. An Ethernet connection is established in order to exchange data between the testing FPGA and a test computer. The protocol used is custom-designed and builds up on top of the Ethernet layer.
-2 - Dedicated configuration and testing software is deployed on the test system control computer. This control software supports test engineers in the configuration process of the GBTX. In figure 2 the configuration panel for the RX and TX data path inside the GBTX is shown. By simply clicking on the corresponding arrows, the user can change the data path, skip blocks or enable loopbacks.
The software sends control signals and values over Ethernet to the FPGA. The FPGA uses the I 2 C interface to program the intended configurations into the GBTX. Testing the GBTX required performing Single Event Upset (SEU) tests [4] in which the device is placed under radiation. A standard measure for communication links is the bit error rate (BER) in which basically the discrepancy between sent and received data is determined. The test data is generated in the FPGA. The optical connections and the E-Links are used to circulate those test data through the GBTX and to receive them back in the FPGA where they can be compared.

The testing and characterization framework
As GBTX tests are highly complex and data demanding, software components were developed to deal with the specific requirements. One of the challenges of SEU testing the GBTX is to understand the causes of errors and to be able to investigate them further. There are many parts of the circuit that can be affected by radiation. To understand which part causes bit errors in each individual situation requires powerful test tools. The most important are shown in this segment.

Database logging
Logging all information sent from the SAT board to the test software is a key functionality of the test framework. There is a wide set of possibilities to accomplish this task. Those range from simple text files to dedicated logging frameworks. The GBTX test software is based on a common DataBase (DB) system. To be precise, it is a CERN maintained Oracle 11g DB.

JINST 10 C03035
One of the challenges of the logging system is to cope with a huge amount of data. Those test results and status information are not being received in a uniform distribution, but can contain significant peaks. Peaks always occur when BER events take place. In that case all the information regarding the affected frames is being captured. It has to be ensured that those are safely stored in the system. A well-approved Relational Database Management System (RDBMS), as found in the used Oracle 11g DB, fulfils this task with high reliability.
A DB is a good solution to support scalability. Scalability is always a key asset in case logging requirements can change and might increase. As soon as a software iteration results in a significantly higher amount of data to be stored, simple text file storage might be hard to adapt. An approved RDBMS handles data transfer with high speed and reliability. It is generally able to help the consuming software to abstract from the details of storing the data.
Using existing DB solutions reduces the development time significantly. Instead of customizing logging solutions, most RDBMS are capable of fulfilling by default the mentioned requirements that the GBTX testing framework poses. One of the key elements of an RDBMS is to handle the concurrent access on a limited resource. Simultaneous read and write accesses are being taken care of by the system, allowing users to analyse data while at the same time more logging information can be stored [5].

Live viewer
Based on the advantages of using a DB for logging the test information, a so called LiveViewer directly analyses and displays the most important information to the test engineers. Like the other components, this tool is written in Java and builds up on the RDBMS dealing with the complexity of concurrent read and write requests and the fast read out of organized data.
Besides showing the logged information directly, test engineers can even interact with some of the analysed data. In figure 3 an example of this is given. The figure highlights bit error events. An engineer can by simply moving the mouse over one of those events see the corresponding wrong frames associated. Those are displayed on the right-hand side starting with the frame index. Concurrently, for each corrupted frame the position of wrong bits is computed, statistics are made and a histogram is built to give an immediate overview of the situation [6].
The powerful attention to detail that the LiveViewer provides allows for very precise and timeefficient testing. This is highly important as radiation tests are often conducted under strict time constraints, being frequently limited to a couple of hours. Figure 4 is a good example for providing many details. It groups the frames according to the amount of wrong bits contained in them. The graph is built based on all wrong frames encountered since the start of a test run.
Those graphs enable test engineers to conduct immediate diagnostics on the part of the ASIC suffering from radiation. Much more information is being computed together with the histograms and plots described above. For example the number of configuration errors automatically corrected by the ASIC, losses of lock of the transceiver and many more. The full control panel of the SEU tests is complex and provides further graphs and information. It is shown in figure 5.

GBTX emulator
In order to verify that the testing software reacts in a correct way under all conditions, a GBTX emulator was used. It allows simulating conditions that are hard or impossible to reproduce and test in a controlled manner in the laboratory.     The occurrence of errors in the GBTX registers represents such an example. The GBTX has more than 300 8-bit registers, which ideally hold their values, unless re-programmed by the testing software. In a radiation test however, upsets occur that force some values to change. All the registers are triplicated [7], meaning that the chip can correct most of the upsets itself. To verify that this works and to correct upsets that could not be corrected, the full register set is periodically read back by the computer. There the test software analyses the values and corrects them under some specific conditions. Since test engineers are not able to upset a specific register in a controlled manner in the laboratory, the GBTX emulator helps verifying that the software reacts well.
The emulator implements the same protocol as the test software and the test board. It is also able to communicate on the Ethernet interface. That way the test software considers the emulator to be the SAT board, allowing the engineers to precisely induce specific events. Those events can be set up with the help of a dedicated user interface.

Production testing
The production testing draws back on the previous testing efforts. Those are being automated and executed sequentially to conduct a most complete testing of each GBTX chip. The SAT board is again used for this part of the testing. The production test software controls the test sequence by sending commands to the FPGA mounted on the SAT. It receives the results over the same protocol and Ethernet connection as used in the characterization phase. As shown in figure 5, some of the data acquired through those tests are stored in the GBT database. At this point starts the life cycle of each individual chip.
Those initial data include the outcome of the test and a unique identification number. The GBTX has two 8-bit test fuses that are used to assign a serial number. As soon as a chip passes the test the next available number is queried from the database. It is then burned into those two registers. Doing that leads to two advantages. On the one hand assigning a serial number allows the support team to track a specific GBTX and reconstruct its performance history. On the other hand it helps verifying that only good chips had been delivered to the users. This is important as errors placing bad chips into batches of good ones can never be completely excluded.

Customization and user configuration
The main user interface to the GBTX is a dedicated project website [8]. As one of its main functions, it enables users to create customized configurations. A wizard guides the user step by step trying to simplify the process as much as possible. The impact of the configuration choices the user makes is calculated in the background. The wizard is completely dynamic and allows the user to choose only values that are possible in the selected scenario. In figure 6 an example of this is shown. The user chose to create a configuration for the Widebus Transceiver (TRX) mode. Based on this choice, in the following steps of the wizard, the tool will only display configuration options that apply in this mode.
Saving a configuration, the user immediately receives a corresponding programming file sent by email. The configuration itself remains saved on the GBT database. Users always have the possibility to review, reload and edit previous configurations.
The programming files users receive by completing a configuration can be programmed to the GBTX using a specific Java Programmer available on the web site. A tutorial on how to use this application and many other questions concerning the chip are dealt with in a specific section made available as well.

Full life cycle tracking
As shown in this paper, for every individual GBTX component, data are aggregated for the entire life cycle. The process begins with the production tests in which unique serial numbers are assigned. Afterwards, reported issues and selected configurations are saved. As illustrated in figure 7, this aggregated data set is the base for a set of services surrounding the GBTX. Support requests can be handled much faster by the technical team thanks to them having all information about the usage of specific chips. They can directly see the configurations of components that are not working properly. Based on that information a knowledge database will be created. It will allow understanding why problems occur, find indicators for problems and possibly even predict issues.

Summary and outlook
As shown, a complex software environment was created that enables characterization, production testing and customization of the GBTX. The paper depicts the challenges posed by the GBTX, mainly being its complexity, as well as the solutions found for each task. It is shown how having -7 - developed a comprehensive software package for the full GBTX lifecycle facilitated each step of this lifecycle, and is expected to continue to do so in the future.
For the time ahead, it will be important to gather and analyse the feedback of users accessing the GBTX configuration site. Based on this feedback, the site will be adapted in order to provide the best possible user experience. It will also be interesting to follow up on the development of the knowledge database created through the life cycle of the chips. It is exciting to understand, whether or not the data gathered will be sufficient to implement automatic measures pointing to common indicators of failures.