Standardization of automated industrial test equipment for mass production of control systems

Power converters and their controls electronics are key elements for the operation of the CERN accelerator complex, having a direct impact on its availability. They must be designed to achieve a high Mean Time Between Failure (MTBF) and hardware reliability must be ensured by board level testing before hardware is assembled and installed. In this framework, the National Instrument PCI extension for Instrumentation (PXI) was chosen as standard platform for the development of testers. This paper reports on the design strategy and approach used focusing on the tester hardware, firmware and software development.

A : Power converters and their controls electronics are key elements for the operation of the CERN accelerator complex, having a direct impact on its availability. They must be designed to achieve a high Mean Time Between Failure (MTBF) and hardware reliability must be ensured by board level testing before hardware is assembled and installed. In this framework, the National Instrument PCI extension for Instrumentation (PXI) was chosen as standard platform for the development of testers. This paper reports on the design strategy and approach used focusing on the tester hardware, firmware and software development.

Introduction
The CERN accelerator complex consists of a series of accelerators that work together to accelerate particles to increasingly higher energies. Several different magnets are used to deflect, focus and defocus the particles during their trajectory. For powering these magnets different power converter topologies are used. The machine overall availability is heavily influenced by the reliability of the power converters and their control electronics. A big challenge for the LHC is to increase the machine availability by ensuring higher Mean Time Between Failure (MTBF) of the accelerator components. A key part of this is to provide complete board level testing before controls hardware is assembled and installed. A key element to achieve this is the development of industrial testers used in the production phases.
The roles of these testers are two-fold: to first validate correctness of assembly during the mass production phase and secondly to provide a means to accurately diagnose failure modes and repair failed electronics brought back from operation during exploitation. The Electrical Power Converter (EPC) group at CERN is in charge of the design, development, installation and maintenance of electrical power converters for all accelerators. In this framework, the Converter Controls Electronics (CCE) section provides the electronic infrastructure for controlling and monitoring those systems. Recently, the National Instrument PCI Extension for Instrumentation (PXI) was chosen as standard platform for the development of testers. PXI is a modular platform for measurement and automation systems offering a low-cost and flexible solution, providing the benefits of an open industry standard. National Instruments provides many standard modules to choose from, each providing a different function. Third party vendors can also produce boards that work with the PXI standard, making the system extremely adaptable to any tester configuration. In comparison with a complete in-house design, this allows the testers to share the same hardware and software structure resulting in significant resources sharing, a quick turnaround time, and a lower cost per-unit. In addition to the PXI crate, the generic test setup consists of a specific Test Control Card (TCC) designed to interface the PXI modules with the Device Under Test (DUT), and a control sequence -1 -written in LabWindows, a C-based language providing an easily maintainable structure. This paper reports the design strategy used for the hardware, firmware and software development and an example of a complete set of testers for the new radiation tolerant version of the Function-Generator Controller (FGClite) that is an electronic module for the converters control. FGC is designed to allow automatic and remote control of power converters. Some of the converters, together with their control systems, are located in areas that are exposed to beam-induced ionizing radiation. The FGCLite was designed to endure these radiation-induced effects, having a direct impact upon the accelerator's availability.

The PXI platform solution
Before moving to the PXI solution, a Generic Test Chassis (GTC) was developed to test electronics. The GTC hardware was complex, difficult to maintain and expensive. Every card type that had to be tested required a specific, complex, test card to adapt its interfaces, to correctly control and observe the various resources, specific to that card. Every test card consisted of at least a microcontroller, an FPGA and a flash memory, all these complicated the design, realisation and debugging of the GTC hardware and software. With the increase in electronics production required for the FGClite project, the EPC/CCE section took the opportunity to move to a new platform.
PXI is a rugged PC-based platform for measurement and automation systems. PXI combines PCI electrical-bus features with the modular, Eurocard packaging of CompactPCI and then adds specialized synchronization buses and key software features. National Instruments provides many standard modules to choose from, each providing a different function. Third party vendors can also produce boards for the NI PXI system making the system extremely adaptable for test setups. This allows the testers to share hardware and software structures, resulting in significant resources optimization which leads to a quicker turnaround time, and overall, a lower per-unit cost than a complete in-house design, as had been done for the GTC.

Hardware architecture
The PXI-1042Q Chassis, with dimensions of 40 cm × 27 cm × 18 cm, can house up to 7 data acquisition and generation modules and a controller card. The controller is running Windows operating system that uses LabWindows CVI as software platform executing test programs that access the PXI bus and its associated modules. The PXI crate accepts many different modules to customize the tester accordingly to the test to perform. Additionally, the system can be easily fitted to bigger versions of the chassis and other controllers, which increases its scalability. Modules of the PXI System used for testers in converter controls electronic are shown on the table 1.
In addition to the PXI crate, the generic test setup consists of a specific Test Control Card (TCC) designed to interface the PXI Chassis with the Device Under Test (DUT). The TCC is by definition the hardware part dedicated to each card or module that must be tested. The TCC is mainly used to interface in terms of voltage level or signal translation, between the specific board to test and the PXI. Sometimes it embeds a specific programmable logic or devices.
The following picture presents typical elements included in a test bench.

Software architecture
Each tester is implemented by developing a control sequence written in LabWindows/CVI, a Cbased environment, providing a maintainable and reusable structure. As all test benches are based on the same software, it is easy to foresee many test benches that share a great number of resources. For this common part of test benches, the same C code template can be used as basis for each new test bench project. Obviously, for each test bench, the dedicated part will be redesigned according to the current system to test. The template provides the ability to communicate with the PXI boards, meaning the DUT can be powered and the DAQ modules can read and write both digital and analogue values using an Excel spread sheet.
An example of a software procedure is shown in the figure 2. At the beginning of a test software development, all test structures and functions validating particular DUTs' parts are declared and initialized.
Before running the test, a list of tests to be executed has to be provided for the software. This is done by loading a previously constructed test list file. Test files cover both board level tests and others designed to exercise sub-sections of board. Test commands are processed from an Excel -3 -  spreadsheet, defining the test steps and limits for each. This provides a user-friendly means of designing and executing LabWindows CVI-based tests. In order to be able to execute functions based in the Excel spreadsheet, a special library that associates strings in Excel Spreadsheet file cells with function names has been implemented in the CVI software. An example of the test sheet format is shown in figure 3.

JINST 11 C01047
A dedicated Graphical User Interface (GUI) is used throughout the tests (see figure 4). Each test function corresponds to its indicator on the screen. At the centre of the panel is a block diagram of the DUT, with virtual LEDs on critical components and signal paths. Beside this, information about the current task and overall test progress is displayed. Once the test has completed, the LEDs turn green when a test has passed or red if the test has failed.
A detailed, unique log file is created at the end of each test for each board. It includes test information such as: date/time, test list performed, a unique identified read out from each board, board barcode, temperatures, measured parameters (if required) and more. An example of the log file is shown in the figure 5. Creation of the log file ends every test procedure; then the user can decide to perform the same procedure again, change it, or finish.

FGClite testers
The LHC has several thousand magnets, both warm and super-conducting, which are supplied with current by power converters.

Conclusions
Over the past years different platforms for automated test were evaluated. Several versions of a Generic Test Chassis (GTC) were developed. It turned out that the complexity of the hardware, software and the excessive cost became too problematic. Therefore, recently, the National Instrument The first principle use case of the test platform is the FGClite project, which is scheduled for mass-production in 2016.