Evaluation of a commercial AdvancedTCA board management controller solution (IPMC)

The MicroTCA (MTCA) and AdvancedTCA (ATCA) industry standards have been selected as the hardware platform for the upgrade of the electronic systems of some of the experiments at the Large Hadron Collider (LHC) . In this context, the electronics support group for experiments at CERN is running a project to perform technical evaluations of MTCA and ATCA equipment. As part of this activity, a commercial solution for an Intelligent Platform Management Controller (IPMC), an essential component of any ATCA blade design, is being evaluated. We validated the supported IPMC features, checked the interoperability and adapted the reference design for use on an existing ATCA carrier board.

. In this context, the electronics support group for experiments at CERN is running a project to perform technical evaluations of MTCA and ATCA equipment. As part of this activity, a commercial solution for an Intelligent Platform Management Controller (IPMC), an essential component of any ATCA blade design, is being evaluated. We validated the supported IPMC features, checked the interoperability and adapted the reference design for use on an existing ATCA carrier board.

K
: Modular electronics; Detector control systems (detector and experiment monitoring and slow-control systems, architecture, hardware, algorithms, databases) 1Corresponding author. This paper presents the IPMC functionality evaluation results, the design of the IPMC adapter card, the outcome of the commercial IPMC mezzanine adaptation to an existing ATCA carrier blade as well as the design of a CERN IPMC mezzanine card.

ATCA standard: Hardware Platform Management
AdvancedTCA (Advanced Telecommunication Computing Architecture) is an open architecture described by a series of standards from PICMG [1]. This platform, mainly used for telecom, medical, military, industrial and physics applications, offers multiple advantages: modularity, interoperability, massive data bandwidth on backplane, power and cooling redundancy as well as a standardized and redundant central management system. The Hardware Platform Management described by the AdvancedTCA standard specifies an interface to monitor sensors (voltages, temperatures, etc.), control the system (power management, port/clock activation, etc.) and ensure proper operations (compatibility between the different boards, hot swap, redundancy, etc.).
The hardware management is performed via modules described by the AdvancedTCA standard. All these modules, listed below, are interconnected via I2C buses. The communication protocol is based on the Intelligent Platform Management Interface (IPMI) [2] standard specified by Intel. Figure 1 introduces the hardware management modules and their interconnections: The shelf manager monitors and controls an entire shelf. This module checks the health of the system and controls the power as well as the cooling of the crate.
Cooling units, also called fan trays are controlled by the shelf manager to provide a sufficient airflow across the blade.

JINST 11 C02044
The ATCA board is the user specific application payload module. The hardware management is locally carried out by an on-board IPMC module. The information is reported to the shelf manager via an IPMB bus based on IPMI over I2C.
The carrier ATCA blade is a specific ATCA board designed to host Advanced Mezzanine Card (AMC). The hardware management is also based on an IPMC module implementing additional features to control the AMCs. The name of this controller is carrier IPMC.
The AMC card is an ATCA mezzanine dedicated to the user application. It can be used with a carrier ATCA blade or in a MicroTCA architecture. The hardware management is locally carried out by a Module Management Controller (MMC). Information is reported to the carrier manager (carrier IPMC for AdvancedTCA).

Commercial IPMC solution
Pigeon Point Systems, recently acquired by Schroff/Pentair, is specialized in the development of the hardware management modules [3] described by the PICMG standard. CERN purchased a license for the complete Hardware and Software solution for the IPMC / Carrier IPMC module to evaluate this commercial solution.
The Pigeon Point IPMC / Carrier IPMC solution is based on the Smart Fusion A2F200/500 FPGA. The following block diagram (figure 2), derived from the IPMC specification, introduces the different features implemented in their starter kit. All the required interfaces to manage the ATCA blade and up to 2 AMCs are implemented (IPMB buses, standardized I/Os, power management, sensor interfaces, etc.). The specification notes that the firmware can manage up to 10 AMCs. In addition, this solution provides serial -3 -

JINST 11 C02044
interfaces for debug and Ethernet with RMCP support to send IPMI commands directly to the IPMC. The starter kit, included in the license package, is based on an interconnection card hosting the Pigeon Point IPMC mezzanine. This setup implements the following features:

Technical evaluation
This section presents the commercial IPMC technical evaluation process, the setups used to assess the module characteristics and finally the results obtained.

Evaluation process and setups
The first evaluation phase consisted of testing the Pigeon Point solution. In parallel to the specification check, the functionalities listed below were tested using the Pigeon Point starter kit: In order to overcome this restriction and test the IPMC in a custom environment, the module was tested in a second evaluation phase with an existing ATCA board. Many custom ATCA blades have been designed featuring a connector compatible with the existing LAPP IPMC mezzanine (DIMM-DDR3 VLP form factor). Hence, an adapter PCB was designed and produced to evaluate the commercial IPMC in a real ATCA environment. The adapter card was designed in order to provide a connectivity compatible with the LAPP IPMC connector pinout ( figure 3).
This adapter card is mainly based on the Pigeon Point IPMC mezzanine connector and a CPLD to allow rerouting of the signals to the DIMM-DDR3 VLP connector (LAPP connector). The block diagram (figure 4) shows its architecture.
Most of the signals present on the Pigeon Point IPMC mezzanine connector are directly routed to the FPGA. Therefore, their assignment and functionality can easily be changed. To take full advantage of this, the IPMC adapter card was designed to be customizable: signal routing from the Pigeon Point IPMC to the LAPP connector can be modified easily using a MAX V CPLD from -4 -

JINST 11 C02044
The IPMC FPGA firmware was modified to implement new drivers for the various non-Pigeon Point supported chips implemented on the LAPP ATCA carrier (sensors, AMC power switches). The AMC Test Board hosting the CERN MMC mezzanine was successfully used to test all AMC management features except the RTM features.

Technical evaluation outcome
The list below shows the features tested with the previously described setup: • Communication with the Shelf Manager

CERN IPMC mezzanine based on the Pigeon Point solution
Following the positive evaluation results obtained with the adapter card, the decision was taken to design a Pigeon Point based IPMC mezzanine compliant with the DIMM-DDR3 VLP form factor. The first challenge was to validate the implementation feasibility on a rather limited area (DIMM-DDR3 VLP). Therefore, a representative mechanical drawing (figure 6) was made with the main required components to verify that everything fits. Due to the limited space available, the smallest FPGA supported by Pigeon Point was selected, forcing the addition of I/O extenders for the AMCs management signals. This solution was successfully tested using the adapter card as described in the section 2.
The A2F200 FPGA design was modified to meet the specification summarized in the following block diagram (figure 7): The layout is now complete and the mezzanine card prototype production will be launched before the end of the year. Figure 8 shows the top view of the IPMC card.
The next step will consist of modifying the IPMC firmware to implement the hardware required features as well as to build a user space for board customization (sensors monitoring, OEM commands, etc.).

Conclusion
The Pigeon Point Hardware and Software IPMC solution was acquired by CERN to perform its technical evaluation. The evaluation of the product was performed using the starter kit included in the license package and a real ATCA environment based on the LAPP carrier board. Following the positive results obtained with the adapter card on the LAPP carrier board, a LAPP-IPMC form factor compliant mezzanine was designed. The schematic and the layout of this mezzanine have been reviewed and a prototype series will be produced before the end of the year. Following the validation of the intelligent-RTM feature and the adaptation of the IPMC firmware (user part for easy customization), the next step will consist of testing this mezzanine on different ATCA blades. Finally, the mezzanine will be tested with multiple blades running in a common shelf (testing possible interoperability conflicts). In case of success, a CERN IPMC hardware and software solution might become available to experiments developing ATCA hardware.