Design of an AdvancedTCA board management controller (IPMC)

The AdvancedTCA (ATCA) standard has been selected as the hardware platform for the upgrade of the back-end electronics of the CMS and ATLAS experiments of the Large Hadron Collider (LHC) . In this context, the electronic systems for experiments group at CERN is running a project to evaluate, specify, design and support xTCA equipment. As part of this project, an Intelligent Platform Management Controller (IPMC) for ATCA blades, based on a commercial solution, has been designed to be used on existing and future ATCA blades. This paper reports on the status of this project presenting the hardware and software developments.


General overview and specification
The AdvancedTCA standard, defined by the PCI Industrial Computer Manufacturer Group (PICMG) [1], outlines a modular architecture by defining physical, electrical and functional specifications like the Hardware Management Platform shown in figure figure 1. It offers a wide range of hardware management features to monitor (temperatures, voltages, current, etc.) and control (fan speed, power management, etc.) the system as well as to ensure its proper operation (modules compatibility, current requirement, e-keying, etc.  -1 -

IPMC
The AdvancedTCA standard describes Hardware Platform Management modules responsible for the control and monitoring of the system. In this context, the IPMC shall provide to the shelf manager the information related to the AdvancedTCA and/or carrier blade: Field-Replaceable Unit (FRU) information (serial number, reference, part number, manufacturer, AMCs/RTMs support, current limit, port description, etc.) and a list of sensors with thresholds (non-recovery, critical, non-critical). According to the shelf capability, the blade is turned ON and the ports connected to the backplane are activated (the handle switch allows the user to request insertion/extraction processes). Additionally, the IPMC shall monitor the sensors (temperature, current. . . ) used to regulate the system (e.g.: cooling) and/or detect failures. In case of threshold detection, the controller automatically sends events to the shelf manager. These events can also be caught by an external monitoring system.
A first technical evaluation of a commercial solution, designed by the Pigeon Point System group [3] was performed in 2015. Following the positive results obtained [4], this platform was used to design a mezzanine card to be proposed to the experiment's system designers.

Specification
The CERN EP-ESE group decided to design a versatile module that can be used by the experiments. To be compatible with existing designs and to limit as much as possible the required blade surface, the very low profile ram memory for factor (DIMM-DDR3 VLP), shown in figure figure 2, was selected for this IPMC mezzanine. According to the ATCA standard, the height of the mezzanine card shall be lower than 18.3mm: components shall have a maximum height of 21.33mm (3mm are used by the IPMC connector). A mechanical drawing, with the smallest FPGA supported by Pigeon Point and all the non-mandatory features removed (e.g.: LPC interface, ADCs. . . ), was used to first prove the feasibility of such a mezzanine.
The mezzanine connectivity and the software/firmware implementation allow the support of the following features:

Design of the mezzanine card
The Pigeon Point schematic was adapted to implement all the features described in the specification section. Due to the limited available area on the DIMM-DDR3 VLP form factor, the smallest supported FPGA was selected: Microsemi A2F200 device in CS288 package (11 × 11 mm) [5]. The fine BGA pitch of 0.5mm resulted in densely routed traces.
A first layout with 10 layers, using buried and blind vias as well as small traces/clearances was designed. That resulted in the stack layer shown in figure figure 4. Following discussions with Printed Circuit Board (PCB) suppliers, it appeared that the via configuration was outside of the standards (IPC-2221) [6], shown in figure figure 5, and outside of the tolerances achievable by the manufacturer. According to the specifications, the maximum layer thickness for blind via of 0.2mm drilled size must be in the 0.10-0.25 mm range but is 0.670 mm in our case.  However, decision was taken to produce the first prototype in order to be able to assess the mezzanine schematic. Due to the difficulty to manufacture it, the PCB production was very expensive. In addition, a few vias cut the annular ring as shown in figure 6, which increases the risk of a via failing during thermal cycling. Therefore, the decision was taken to correct the layout and make the design rules compliant with the manufacturer DFM guidelines.
The new layout, which improves the component placement and the routing, results in an 8-layer PCB with only through-hole vias. With the successful production (figure figure 7) and testing of the CERN IPMC prototype board, the hardware design is complete.

Firmware
The Pigeon Firmware was modified to implement all the features described by the CERN specification. The block diagram presented within figure figure 8 shows the modules implemented to make the IPMC fully functional.  Following a first evaluation of the master JTAG interface based on GPIOs, an additional HDL module was made to accelerate this feature. This VHDL IP, shown in figure 9, is a simple shift register clocked by the 20 MHz clock provided by the MSS_CORE. The APB bus, coming from the ARM controller, is used to write the bytes to be sent via JTAG.

Software
The software is written for the FPGA embedded cortex-M3 processor using Softconsole v3.4 from Microsemi (based on Eclipse). The architecture of the project contains multiple folders, each one dedicated to a specific function. All of the software features can be configured using header files. It allows the user to easily customize the IPMC controller to meet their requirements: sensors, AMC management, power sequences, etc. Finally, the FRU/SDR information is generated using additional python scripts and configuration files written by the user (fru-info.inf and sdr-info.inf).
A major modification was made to implement the TCP/IP protocol and make the Ethernet polling faster. These modifications were required to design the Xilinx Virtual Cable deamon that allows configuring and debugging Xilinx FPGA located on the ATCA blade via the IPMC controller.

Status and roadmap
The first prototypes of the IPMC mezzanine card were produced and used to demonstrate the feasibility to adapt the Pigeon Point solution to our requirements. The final IPMC prototype was validated using real AdvancedTCA systems made up of a Schroff crate, the Pulsar IIb ATCA blade and the AT8901 Kontron switch blade. The following features were successfully tested: • Management of the AdvanceTCA blade.
-Sensors monitoring and event generation.
• Management of up to 4 AMCs.
-AMC detection and hot swap insertion/extraction were successfully tested using the CERN MMC.
-AMC sensor event management.
-IPMI support over the Ethernet interface.

JINST 12 C03010
The CERN IPMC supports Xilinx FPGA configuration using a TCP/IP server based on the open source Xilinx Virtual Cable solution. Timing measurements were made to estimate the time required to configure a Virtex 7 FPGA as shown in figure figure 10. The hardware is ready and a first production of 50 cards will be launched to provide a first mezzanine to the early users. Meanwhile, a system being setup, with a remote computer located at CERN, to allow users to customize the Software. The next steps of the project are: • Implement TAP server for non-Xilinx devices configuration using JTAG.
• Develop an IPMC test bench.
• Launch first production of 50 pieces.
• Setup a development environment for the users.
• Write and provide a detailed user guide.
The final kit, including the Pigeon Point royalty, will be available for users in Q1/Q2 2017.