Evaluation results of xTCA equipment for HEP experiments at CERN

The MicroTCA and AdvancedTCA industry standards are candidate modular electronic platforms for the upgrade of the current generation of high energy physics experiments. The PH-ESE group at CERN launched in 2011 the xTCA evaluation project with the aim of performing technical evaluations and eventually providing support for commercially available components. Different devices from different vendors have been acquired, evaluated and interoperability tests have been performed. This paper presents the test procedures and facilities that have been developed and focuses on the evaluation results including electrical, thermal and interoperability aspects.


Overview
The Telecommunication Computing Architecture (xTCA) is a series of specifications defined by the PCI Industrial Computer Manufacturer Group (PICMG) including AdvancedTCA (ATCA) [1], Advanced Mezzanine Card (AMC) [2] and MicroTCA (MTCA) [3]. These specifications define a standard modular architecture by establishing physical, electrical and functional specifications and ensuring interoperability. xTCA offers a wide range of form factors and allows different levels of redundancy for power distribution, cooling system and shelf management. This makes xTCA an interesting platform for a wide range of applications such as Military/Aerospace, Communications, Medical and Industrial. In 2011 the PICMG consortium, driven by the physics research community, released the MTCA.4 standard as a complementary specification to MicroTCA. This specification defines the guidelines for the implementation of Rear Transmission Modules (RTMs) and provides precision timing for data acquisition electronics. Several independent groups at CERN and in external institutes have started to develop ATCA and MicroTCA modules and the question arises as to how these modules should eventually be housed.
In this framework, the CERN Electronic Systems for Experiments group launched the xTCA Evaluation Project with the goal of providing technical evaluation of xTCA systems with a clear focus on the infrastructure equipment such as shelves, power supplies, power modules, cooling modules and MCHs. The project includes electrical evaluation of power modules, thermal characterization of shelves and IPMI (Intelligent Platform Management Interface) functionality tests. The electrical evaluation of the power modules includes static and dynamic regulation tests, efficiency and power factor measurements, ripple and noise characterization as well as overcurrent protection test. The thermal tests aim at estimating the cooling unit performance and airflow homogeneity inside a shelf. The IPMI functionalities of commercial equipment have been tested using a commercial automated test suite for checking the Hardware Platform Management Software and IPMC -1 - During the test phases, several interoperability problems and technical issues have been uncovered and addressed by working in close collaboration with the manufacturers. This allowed us to acquire knowledge and experience with these new architectures. For each component a detailed evaluation report has been written. A number of reports are already available and can be helpful for selecting components for building an xTCA system. This paper shows the test procedures and facilities used and reports the evaluation results with a clear focus on the electrical, thermal and interoperability aspects of the tested xTCA equipment.

MicroTCA evaluation
The MicroTCA industry standard is gaining momentum among High Energy Physics experiments. At CERN, the first MicroTCA systems are going to be installed during the current Long Shutdown (LS1) while deployment of larger quantities of systems are planned for the Long Shutdowns 2 (2018) [4]. In this framework, the evaluation project aims to evaluate commercially available Mi-croTCA components that can be potentially deployed by the experiments. The evaluation includes tests of Power Modules (PM), shelves and MicroTCA Carrier Hubs (MCH). Moreover, attention is given to interoperability between different components and manufacturers. Table 1 lists the tested components.
-2 -  Figure 1 shows a schematic view of the test setup architecture. A Labview interface based on IPMITOOL [5] generates IPMI commands according to the test being performed. IPMI packets are then sent over the LAN to the target MCH that de-encapsulates and sends the original IPMI command to the target FRU (Field Replaceable Unit e.g. PM, CU). AMC (Advanced Mezzanine Card) and RTM (Rear Transmission Module) load modules controlled through IPMI have been developed in-house in order to perform electrical and thermal tests. They are based on switched resistive loads and provide on board current and temperature monitoring. The components under test were loaded to be able to test them under stress.

Power module testing
In MicroTCA systems the Power Module (PM) provides power and related management features to all the components in the shelf. It takes the input supply and converts it to two voltage levels, 3.3 V for the Management Power and 12 V for the Payload Power. The evaluation of power modules includes static and dynamic regulation tests, efficiency and power factor measurements, over current protection tests as well as ripple and noise measurements. Moreover, management features are also evaluated. Some of the tests and measurements, such as ripple and noise, dynamic regulation and overcurrent protection, were performed in order to verify the equipment behaviour beyond the xTCA standard technical requirements. In order to reduce the influence of the external components such as cooling units, a shared power topology was used. In such a topology more than one PM is defined as primary module delivering power to a defined set of components. For test purposes, an auxiliary power module was used to power the cooling units and the MCH, while the PM under test was used to power the AMC slots (load modules).
Four different power modules from three different manufacturers have been evaluated. Although the modules worked, during the tests several non-conformities with the product specification have been found. Efficiency, output power and input voltage range were the main areas of concern. The issues were reported back to the manufacturers who introduced hardware and software modifications leading to a significant improvement of the electrical characteristics. As an example, table 2 shows the evaluation results of two different hardware versions of a commercial -3 -

MTCA shelf evaluation
The shelf is the mechanical structure hosting the modules and providing cooling by means of cooling units. The MTCA shelf evaluation includes cooling performance, mechanical layout and backplane alignment. The evaluation of the cooling performance is an important aspect since the cooling system should be capable of providing the required airflow specified by the module manufacturers in order to assure their correct operation. For this purpose, the airflow homogeneity and cooling capability was evaluated for each slot by using the AMC and RTM load modules. The board temperatures were monitored using six temperature sensors and the data acquired was then computed to calculate three temperature gradients for each load module. Figure 2 shows the temperature sensors position and the relative vertical temperature differences. In order to distribute the heat evenly across the boards, the load modules were configured to dissipate their maximum power. This resulted in dissipating 90 W on the AMC slots and 40 W on the RTM slots. Furthermore, the board being measured was surrounded on each side by three other boards dissipating the same power.    Table 3 shows the cooling performance of three different shelves. For each shelf, the plot shows the temperature difference averaged for each slot position. Significant performance variations can be noticed depending on the model and the number of fans used.

MCH and interoperability tests
The MCH is the device providing management and data switching in a MicroTCA system. It controls device power activation, system temperatures, fan speed and it handles events such as over-temperature. It communicates with all the devices inside the shelf by means of the IPMI, a standardized message-based interface widely used for management tasks [6]. Most of the interoperability problems occurring in an xTCA system are related to the IPMI communication between the various components. For this reason, the test of IPMI conformity is a crucial item in the MicroTCA evaluation. To perform such tests a fully automated test suite developed by Polaris Networks [7] has been used to test MCHs as well as cooling units, backplanes and AMCs. The software was very useful to predict some interoperability issues related to the module non-conformity with the xTCA specification.

ATCA evaluation
At CERN, ATCA is being adopted for the upgrade of some experiments (e.g. ATLAS and LHCb). The evaluation of ATCA systems includes the shelf cooling performance, IPMI conformity, mechanical robustness and layout and power capability. So far two ATCA shelves have been evaluated (table 4).

Cooling performance evaluation
Similarly to the evaluation of the MicroTCA cooling performance, the purpose of this test is to evaluate the cooling homogeneity across the slots. For this purpose, commercial load modules have been acquired [8]. These boards provide on-board temperature monitoring and can be controlled via IPMI. An automated Labview interface has been developed in order to configure the load modules and read the sensor values.
The test has been carried out dissipating 250W on the front boards and 50W on the rear modules while monitoring the board temperature. Three vertical temperature differences were computed on the front side while only one was calculated on the RTM side. As shown in table 5 the two shelves differ in cooling performance. This mainly depends on the fact that the first shelf consists of a single cooling unit located on the upper side. Furthermore, the number of fans used and their specified air displacement capability have a significant impact on the cooling performance.

Conclusions
As part of the xTCA evaluation project, several MicroTCA and ATCA components have been acquired and evaluated. A set of test tools consisting of load modules and an automated Labview test interface have been developed and built. Electrical tests of power modules have been performed. This included static and dynamic regulation tests, efficiency and power factor measurements, ripple and noise characterization as well as overcurrent protection test. The cooling performance and mechanical layout of MicroTCA and ATCA shelves have been evaluated. Furthermore, the IPMI functionalities and the interoperability of xTCA equipment have been investigated. Although the power modules worked properly, during the evaluation phases several non-conformities with the technical specification have been uncovered. The collaboration with manufacturers has been effective to address equipment non-conformities and technical problems. This allowed us to acquire significant knowledge about the xTCA specifications to build test setups and procedures, and to develop a robust equipment operation know-how. This will guide us towards the next project phase which will consist in defining technical requirements in view of future equipment purchases, make equipment recommendations and provide tools and support for these items.