MicroTCA and AdvancedTCA equipment evaluation and developments for LHC experiments

The MicroTCA (MTCA) and AdvancedTCA (ATCA) industry standards have been selected as the platform for many of the current and planned upgrades of the off-detector electronic systems of two of the LHC experiments at CERN. We present a status update from an ongoing project to evaluate commercial MTCA and ATCA components with particular emphasis on infrastructure equipment such as shelves and power-supplies. Shelves customized for use in the existing LHC rack infrastructure have been tested, and electrical and cooling measurements and simulations were performed. In-house developments for hardware platform management will also be shown.


Introduction
Originally developed for the telecommunication industry, the MicroTCA (MTCA) [1] and Ad-vancedTCA (ATCA) [2] modular electronics standards have been selected as the platform for many of the on-going and planned upgrades of the off-detector trigger and data-acquisition systems of the ATLAS and CMS experiments at the LHC at CERN. In this context the support group for the electronic systems of the experiments at CERN (PH-ESE) has been running a project to evaluate selected commercial MTCA and ATCA components from different vendors with the goal to provide equipment recommendations and eventually support to the users of these modular electronics standards. The emphasis was placed on infrastructure components such as shelves and power-supplies as well as module management hardware and software. Throughout the project, commercial MTCA and ATCA equipment from various vendors has been acquired, evaluated and interoperability tests have been performed. Test procedures have been defined for the different types of equipment and the necessary test tools have been developed in-house where required. Results from the equipment evaluation campaign concerning MTCA shelves, power modules and MTCA carrier hubs (MCH) as well as ATCA shelves have been reported previously [3] and test reports for the equipment evaluated are available.
More recently, specifications for a custom shelf and power module have been developed in an attempt to standardize MTCA infrastructure components for use in the LHC experiments. Pre-series units of these components have now been qualified and are presented in this paper. In addition, an ATCA shelf customized for use in the existing LHC rack infrastructure has been tested, and results from shelf cooling measurements and rack thermal simulations are shown. Finally, an overview of in-house developments for hardware platform management will be given.

MTCA equipment specification and qualification testing
Taking the specific requirements of the LHC experiments into account and building on the experience and the know-how gained during the equipment evaluation phase of the project, technical specifications for a common MTCA shelf power module have been defined in view of equipment procurement. A price enquiry was launched in 2015, compatible equipment from different vendors was tested and a selection was made. Finally three pre-series shelves and power-modules were acquired and qualified.

Custom MTCA shelf
A custom MTCA shelf was developed specifically for the requirements of the LHC experiments. The shelf design was modified for vertical airflow in order to be compatible with the thermal management infrastructure used in the experiment counting rooms, where the racks feature a recirculating vertical bottom-up airflow. Other key specifications of the shelf are listed below: • 19" rack mount with a height of 7U; • Vertical cooling with 80W per slot; • 12 full-size, double width AMC slots;  The qualification tests performed with the three pre-series units from the selected supplier included electrical (e.g. backplane voltage drop) and cooling measurements, mechanical checks -2 - as well as IMPI compliance tests. Results from the cooling performance evaluation are shown here as an example. This test aims at estimating the cooling unit (CU) capability and the airflow homogeneity inside the shelf. For this purpose, custom AMC and RTM load modules designed in-house have been used to dissipate the maximum allowable power. The test was performed with the fans operating at their maximum speed. The load board temperatures were monitored using six on-board temperature sensors and the data acquired was used to calculate three temperature gradients for each load module. Figure 2 shows the temperature differences between the bottom and top sensor for each of the 12 slots as well as the position of the temperature sensors on the AMC load board. The pre-series shelves were found to be compliant with the specification and purchase orders for more units will be possible beginning of 2016.

MTCA power module
The key requirements for the power module are listed below: • DC/DC power module (−48 V to 12V) • Total (management power + payload power) output power 840W • Input voltage range −40 V to −60 V • Single-width, full-size • Total efficiency above 40% of maximum load: 90% minimum • Support for 16 output channels: 12 AMCs, 2 CUs, 2 MCHs • Management and payload power current monitoring via IPMI • Support for N+1, 2+2 redundancy as well as shared load topologies Three pre-series units from the selected supplier were qualified and were found to conform to the technical specification. The electrical qualification tests performed include line and load regulation tests, efficiency measurements, ripple and noise measurements as well as sensor accuracy and protection threshold tests. In addition IPMI compliance test were performed. As an example results -3 -  from the efficiency measurement of the three pre-series units are shown here in figure 3 and it can be seen that the efficiency of the PM is above 90% for loads over 340W (40% of the maximum output power of 840W) as required by the specification.

ATCA shelf cooling performance evaluation
Several standard commercial ATCA shelves with front-to-back airflow were initially evaluated [3]. A customized version of an ATCA shelf with vertical airflow was then developed by one of the suppliers in order to adapt it to existing thermal management infrastructure of the racks in the LHC experiments and also evaluated.
Results from the cooling performance of the customized shelf are shown below. Commercial load blades have been used in order to dissipate 300W per slot (250W for the front board and 50W for the RTM) and to monitor the board temperatures at several points. A picture of a load blade with the location of the temperature sensors is shown in figure 4 on the right. Figure 4 also shows the vertical temperature difference between the bottom and top load blade sensors as a function of the slot position, the fans were driven at maximum speed during the test. The results show a satisfactory cooling performance with a rather uniform temperature distribution over the slots, validating the custom design.
The above measurements were performed with a single shelf in an open rack in a lab environment, however the racks in the underground counting rooms of the LHC experiments are based on a closed circuit with heat exchangers to remove the dissipated power. In addition a rack in the experiment could house up to two ATCA shelves. In order to assess the effect of the rack -4 -2016 JINST 11 C02022 environment, the manufacturer of the custom shelf was mandated to perform detailed thermal and airflow simulations of a full rack. Different configurations were simulated and figure 5 shows an example of the simulation results with temperature and air speed within a rack with two fully loaded ATCA shelves dissipating 400W per slot and three 1U three heat exchangers. As can be seen in the temperature plot on the left, the air temperature stays below 35˚C in the rack. The air speed simulation results on the right shows a rather uniform distribution across the slots.
In order to validate the simulation, a program of rack cooling measurements was also launched in collaboration with a team from the ATLAS experiment. Measurements performed in a rack in the ATLAS underground counting room with one ATCA shelf loaded showed a discrepancy with the simulation results, which required tuning down the simulated air flow to obtain the same air temperatures in the simulation model as in the measurement setup.
Additional load blades have recently been ordered to be able to fully load two ATCA shelves in the measurement setup and further measurements will have to be performed in order to check the in-rack air temperatures and the cooling capacity of the heat exchangers in this worst case configuration. Finally, the possibility of using standard ATCA shelves with horizontal cooling is also being studied, however this solution requires more important modifications to the existing rack infrastructure.

Module management developments
The MTCA and ATCA standards define an extensive set of hardware management features in order to monitor temperatures, voltages, currents and to control fan speed, power management as well as to ensure proper operation, e.g. by checking module compatibility and current requirement, etc. These functions rely on specific controller modules on the plug-in cards which are interconnected via an Intelligent Platform Management Interface [4] (IPMI) bus: the Module Management Controller (MMC) for AMCs and the Intelligent Platform Management Controller (IPMC) for ATCA blades. The latest version of the MMC software developed as part of the xTCA evaluation project as well as the status of the evaluation of a commercial IPMC solution are presented in the following sections.

MMC software
In the framework of the xTCA evaluation project, a Module Management Controller (MMC) has been adopted from developments made at DESY (Deutsches Elektronen-Synchrotron) and CPPM (Centre de Physique des Particules de Marseille). The MMC design is based on an ATmega128L microcontroller from Atmel. The module is implemented as a small mezzanine card as shown in figure 6, but the few components required can also be easily integrated directly on an AMC board.
Recently the firmware of the existing Module Management Controller (MMC) has been significantly improved and extended to include new features such as HPM.1 support for remote firmware upgrade. The code was also restructured and largely rewritten in order to improve the compatibility with the MTCA specification and to simplify the user customization required for adapting the code to a specific Advanced Mezzanine Card (AMC). The MMC now passes all the relevant tests of the commercial software suite for IPMI standards compliancy checking that is used as part of the MTCA equipment evaluation procedure. The latest MMC software version has also been successfully tested on in-house and externally designed AMC modules.
The MMC is being successfully used in several projects and the latest software release and documentation are available [6]. The MMC software has also been ported to the Atmel AT32UC3A3256 32-bit microcontroller, which is used on AMC modules by the CMS experiment, and a new release will be made available after testing has been completed.

IPMC evaluation
A commercial package from Pigeon Point Systems [7] including all the required hardware and software to design a carrier IPMC to be used on ATCA blades has been acquired and evaluated. The IPMC solution supports all the features required by the ATCA standard such as blade, AMC and RTM management, power management, sensor monitoring and standardized remote firmware upgrade capability (HPM.1, HPM.2 and HPM.3). These features have all been tested successfully using an evaluation kit provided as part of the commercial IPMC solution.
In addition, an adapter card has been designed in order to be able to connect the IPMC mezzanine provided as part of the evaluation kit on an existing ATCA blade designed to be used with the IPMC from LAPP (Laboratoire d"Annecy-le-Vieux de Physique des Particules) [8]. Figure 7 shows a picture of the test setup with the adapter card carrying the PigeonPoint IPMC installed onto an ATCA test blade.
All essential features such as blade and AMC management, sensor reading, etc. as well as IPMI standards compliance were successfully validated in this setup. Finally an IPMC mezzanine in the same form factor and pinout as the LAPP IPMC (VLP DDR3 DIMM) has been developed and is currently being fabricated. More details on the IPMC evaluation can be found in [9].

Summary
In an attempt to standardize MTCA equipment for use in the LHC experiments, specifications for an MTCA shelf and a power module have been developed, different models have been evaluated and a selection was performed. The MTCA shelf is a custom design suitable for the vertical airflow used in the existing rack infrastructure of the LHC experiments. Pre-series production units of these components were qualified and a purchasing framework was put in place.
A prototype ATCA shelf customized for vertical airflow has been evaluated and cooling measurements were performed. In addition detailed thermal and airflow simulations of these shelves installed inside an LHC rack have been carried out and the results are encouraging. Finally cooling measurements in a real LHC rack environment have also been performed by the ATLAS experiment, however further work will be required to validate a configuration with two fully loaded ATCA shelves inside a rack.
The firmware of an existing Module Management Controller (MMC) has been significantly improved and extended to include features such as remote firmware upgrade as well as simplifying -7 -the user customization required for adapting the code for a specific Advanced Mezzanine Card (AMC). In addition a commercial package including all the required hardware and software to design an Intelligent Platform Management Controller (IPMC) to be used on ATCA blades has been acquired and evaluated. It has been successfully tested on an existing ATCA blade using a custom adapter card. In addition, an IPMC mezzanine based on this design was implemented in the LAPP IPMC form factor and a prototype is currently in production.