Design and Implementation of Detector Control System for Muon Forward Tracker at ALICE

ALICE is the experiment at the CERN LHC devoted to study heavy-ion collisions. An upgrade program of the ALICE detector is ongoing toward the LHC Run 3 starting in 2021 together with the upgrade of the data acquisition system and the detector control system (DCS). One of the main projects of the current ALICE upgrade program is the addition of the muon forward tracker (MFT), a new silicon pixel detector located at forward rapidity. In this paper, we describe the DCS of the MFT detector which is entirely controlled via a finite state machine in a hierarchical system.

integrated onlineâĂŞoffline system [8], and the addition of the muon forward tracker (MFT) [6,9], a silicon pixel tracker at forward rapidity.

Muon Forward Tracker
Heavy quarks, charm (c) and bottom (b) quarks, are known as good probes to investigate the characteristics of the QGP. The muon spectrometer of ALICE [10,11] has performed successful measurements of the J/ψ production rate in the forward rapidity region during Runs 1 and 2 in various collision systems from pp, p-Pb to Pb-Pb [2]. The separation of J/ψ from B hadron decay from prompt J/ψ has been, however, impossible due to the presence of a hadron absorber of 60 radiation lengths placed between the interaction point and the muon tracking system. It induces large multiple scatterings and limited spatial resolution around the interaction point, preventing the determination of the origin of muons. In order to overcome this limitation, a new silicon pixel detector, the MFT, is installed between the interaction point and the hadron absorber. It covers the forward pseudo-rapidity range of -3.6 < η < -2.5 to match most of the muon spectrometer acceptance. The pointing accuracy of the muon production point is consequently improved by matching the tracks measured by the MFT and by the muon spectrometer. The CMOS monolithic active pixel sensor (CMOS-MAPS) technology has been chosen for the sensors. The adopted sensor is developed for both the new ALICE inner tracking system (ITS) and the MFT, and is called the ALICE pixel detector (ALPIDE) [12,13]. The dimension of ALPIDE is 1.5 × 3 cm 2 with the pixel pitch of 27 × 27 µm 2 . It has a spatial resolution of about 5 µm and the charge integration time of 30 µs. A total of 936 ALPIDE chips are used for the MFT covering about 0.4 m 2 . Figure 1 shows the 3-dimensional view of the full MFT. The MFT is separated into two half cones, called the top and the bottom MFT's, respectively. A half cone is composed of five half disks, called half disk 0 to 4, each with two detection half planes. Each detection half plane is split into four zones, each of which is commonly powered and read out in order to reduce the number of connection lines. Figure 2 shows the definition of zones of half disk 4 as an example. A zone corresponds to a set of three to five sensor ladders connected to a single readout unit board (RU). Each ladder, housing two to five ALPIDE chips, is a flexible printed circuit on which the sensors are wire-connected. The ladders are glued on the support planes and connected to printed circuit boards allowing the power and data connection. A total of 280 ladders composes the full MFT.

Online-Offline Computing System
Continuous readout of raw data without any trigger and subsequent simultaneous data processing are a challenge of the ALICE upgrade program. A typical data volume produced by the ALICE sub-detectors will be 3.4 TB/s in Pb-Pb collisions at √ s NN = 5.5 TeV at a collision rate of 50 kHz.
ALICE hence develops a new computing system, named the online-offline computing system (O 2 ), in which online and offline systems are merged in a single operating system [8]. The data links between the acquisition system and the front end electronics (FEEs) use the gigabit transceiver (GBT) [14] technology developed at CERN. In the O 2 farm, the common readout unit (CRU) on the first level processor (FLP) splits raw data from the sub-detectors into physics data and slow control data. Figure 3 shows the schematics of the O 2 raw data stream. The slow control data for each event are one of the ingredients for  the data reduction on the FLP. Collected physics data on the FLP are transferred to the event processor nodes (EPNs) with a data rate of 500 GB/s. The EPN processes the raw data online while performing reconstruction tasks such as clustering and tracking. The output of these tasks is transmitted to the data storage at a rate of 90 GB/s.

Detector Control System
The detector control system (DCS) of ALICE is upgraded to follow the O 2 strategy. The control of FEE employs the GBT slow control adapter (GBT-SCA) [15], which is an ASIC designed for slow control in the framework of the GBT. A specific software framework is developed in order to interface the detector FEE to the user interface based on WinCC Open Architecture (WinCC OA). This framework includes two major elements: the ALICE low-level front-end (ALF) running on the FLP and the front-end device (FRED) [16] as shown in Fig. 4. ALF provides low-level access to the CRU links, while FRED translates high-level words from WinCC OA into low-level words consisting of sequences of hexadecimal commands for GBT-SCA operation and accessing the FEE. Slow control data including detector and environmental conditions are handled in the same packet with physics data as the raw data at the FEE. The CRU split the raw data into physics and slow control data. The slow control data come up to WinCC OA via ALF and FRED, while the physics data are transferred to the O 2 farm as described above. The communication protocol between WinCC OA, FRED, and ALF is a distributed information management system (DIM) [17] developed at CERN. A useful framework providing software tools for DCS development on WinCC OA, named the joint control project (JCOP), is also developed at CERN and employed in the ALICE DCS. The monitoring data are collected via the same links in the other way, as shown with the blue arrows. A part of DCS data that are temperatures of ALPIDEs share the same packets with the physics data from the MFT to the CRU. The physics data are sent to the O 2 farm after splitting from the DCS data at the CRU, while the DCS data come to WinCC OA via Ethernet, as shown with the red arrows.

Hardware Structure
The MFT DCS controls and monitors three subsystems as shown in Fig. 5: the low voltage power supplies, the detector and readout modules, and the cooling system.

Low Voltage System
The ALPIDE chip is powered by two voltage lines at 1.8 V, for the analog and digital parts. In addition, a reverse bias voltage up to -3 V can be applied to the ALPIDE sensor to increase its efficiency and cope with the performance degradation induced by the radiation dose. A dedicated board named the power supply unit (PSU) is installed inside the detector, between half disks 3 and 4, to provide local voltage generation in order to avoid large voltage drops in the power cables from the CAEN power supplies to the detector, located about 40 m apart. The PSU board houses DC-DC converters providing the 1.8 V outputs to the detector as well as the GBT-SCA chips to control and monitor the PSU via the CRU.
Power supply modules manufactured by CAEN are used to supply the low voltage (LV). Figure 6 shows the structure of the LV system of the MFT. WinCC OA connects with a SY4527 mainframe using open platform communications (OPC) via Ethernet. Two branch controllers A1676A in the SY4527 communicate with two power supply systems, one to power the PSUs and the other to power the FEE cards, named the readout units (RUs). These systems are based on the CAEN embedded assembly system (EASY) which is tolerant to radiation and magnetic field. Twelve A3009 power supply boards and three A3006 boards are installed in four EASY3000 crates, which are powered by two A3486 modules to convert 3-phase AC to 48 V DC.

Detector and Readout Modules
The RU, a FPGA based system, is employed as the FEE card of the MFT to read raw data from and send configuration to the ALPIDE chips. One RU board reads out raw data from a zone of a detection half plane. A total of 80 RUs composes the MFT FEE system. The FPGA on the RU is used for configuration and operation of the ALPIDE chips. The RU and the CRU communicate through GBT links.

Cooling System
A leakless water system is used to ensure proper cooling of the detector and the RUs. The structure of the system is shown in Fig. 7. A nominal pressure value of the cooling water is set at 0.3 bar,

Logical Structure
A logical tree structure, which describes all devices in operation and monitoring, is designed based on the hardware structure of the MFT DCS. On the one hand, all hardware devices are referred as elements of the tree, and control commands are published from the elements to the devices. On the other hand, the hardware tree on WinCC OA describes how all hardware devices are wired up. Figure 8 shows the relation between the logical tree and the hardware tree.
Alias names are given for the devices that are used in operation. An alias name is reassigned to point a different hardware channel, for example when a low voltage power supply channel has a problem. The hardware elements are implemented on the logical tree by the alias names.

Finite State Machine
The finite state machine (FSM) is a hierarchical control system based on the logical tree structure and state diagrams. It is implemented in the JCOP framework for the MFT detector. Figure 9 shows the FSM structure of the MFT DCS. A control unit is a conceptual part of the detector system and a device unit corresponds to a real device controlled by the FSM. The FSM allows modeling of behaviors of the elements with state diagrams.
State diagrams are defined for all the FSM nodes. Figure 10 shows the diagram for the top node, named MFT in Fig. 9, as an example. A series of actions allows switching on/off and configuring the different parts of the detector system according to a given sequence. Figure 11 is the synchronization table which shows how the top node state is defined based on the states of the two daughter nodes, corresponding to the related subsystems. The only state which allows physics data acquisition is the READY state where all subsystems are on and configured. Other states include intermediate states used either to bring up/down the detector between the OFF and READY states, and secured states for special conditions of the LHC beam (e.g. magnets ramping, beam tuning, beam injection, and beam dumping) or of the ALICE experiment (e.g. changing magnet conditions). The state of the top node moves from any state to the ERROR state whenever a problem arises.
Operation The FSM transfers slow control commands to the ALPIDE chips, the PSU, and the RU via the DCS data line. Commands are sent from the FSM also to the CAEN power supply modules  via the EASY bus. The upper nodes of the FSM send operation commands to their daughter nodes, then the lowest control units transmit the operation commands to the real devices, which the device units correspond to. In the other direction, condition data from the ALPIDE chips and the RU pass through the DCS data line to the FSM.

Software Interlock
The FSM is employed as the software interlock system. The FSM monitors the temperatures of the GBT-SCA, the half planes, the RU, the FPGA on the RU, the mezzanine boards of the PSU, and the ALPIDE chips. When one of the monitored temperatures exceeds a given threshold, the FSM turns off the corresponding channel of the CAEN LV module. The flow and humidity of the cooling air and the temperature of the half disks are also monitored by the FSM. All the LV power supply channels for the entire MFT detector are turned off if the FSM detects an abnormal condition of the cooling air or an excessive temperature of a half disk, The FSM also takes care of communication control to the CAEN mainframe. The software interlock refreshes the OPC server for the CAEN modules when the communication between the CAEN mainframe and WinCC OA is lost. Communication loss between WinCC OA and FRED and/or FLP is another case to trigger the software interlock.

Detector Safety System
Detector Safety System (DSS) is the hardware interlock system based on Programmable Logic Controllers (PLCs), commonly used for all sub-detectors in ALICE. Actions are implemented on the DSS, detector by detector. Figure 12 shows the general layout of the MFT DSS. The DSS turns off all channels of the CAEN low voltage modules for the MFT detector if a crucial problem occurs.
Minor issues, e.g. an excessive temperature in the MFT, or a communication loss with the main power supply, are normally handled by raising an alarm and are shown in the FSM. In these cases, the DSS represents a redundant safety system, being operational even if the communication between the WinCC OA and the CAEN main frames is lost. Any issue in the cooling system should be handled by the DSS. If the cooling system does not work correctly, the sensors on the cooling system give a trigger to the DSS.

Ladder Smoke Test
A quality assurance system of the ladders is set up during the ladder production phase of the MFT project. This system also serves as a basis to develop the first elements of the global MFT DCS system. The first step of the MFT ladder qualification is to power it. This test is named a smoke test. The voltages provided to the analog and digital parts of the chips are ramped up from 0 V to the nominal value of 1.8 V by steps of 0.1 V. The current consumption is recorded and any abnormal power consumption triggers a voltage shutdown. The smoke test detects some defects of the ladders, in both conditions with and without the back-bias voltage.
Setup Figure 13 shows the setup of the smoke test bench. It consists of WinCC OA including the JCOP framework, the CAEN power supply system, and intermediate boards to adapt the cable connection to the ladder. An A1516B low voltage power supply board is inside an SY4527 crate. It supplies voltages in the required ranges of 0.0-1.8 V for analog and digital lines and of -3.0-0.0 V for the back-bias. WinCC OA controls the power supply system and records the test results. A graphical user interface (GUI) has been designed and implemented on WinCC OA. Operators can set the demanded values of output voltages, the numbers of steps from 0.0 V to the required, and the ramping up speed in each of the steps. The test results are recorded in CSV files and screen shots of the GUI in png files as the logs. A safety system is implemented to protect the ladder. The test is stopped when the current exceeds a given threshold or if the GUI is closed. Achievements The smoke test needs to be performed for about 500 ladders including spares. Figure 14 shows the test results of one of the ladders, numbered 2010. A test is performed first without the back-bias voltage. The current consumption of the analog and digital lines reaches a constant value at about 20 mA each which is below the maximum allowed value of 40 mA. The negative back-bias voltage is then slowly increased from 0.0 to -3.0 V keeping analog and digital voltages at the nominal values. The power consumption of the analog and digital lines should stay constant and the back-bias current should not exceed 20 mA. In this example, the smoke test is successful and the ladder is made available for the next steps of the qualification procedure before being used for the MFT detector assembly.

MFT Surface Commissioning
The MFT detector is commissioned at CERN, following production and assembly of its components. The DCS is integrated to the detector system in this commissioning phase in order to test its all functionalities. This stage is called the surface commissioning, i.e. before the MFT installation in the cavern where the ALICE detector is located. Extensive readout tests are conducted during this stage. The MFT DCS is crucial in this phase to ensure safe and easy operation of the detector.
The MFT DCS consists of three separate subsystems as described in Sec. 2. The entire MFT detector system, except the cooling part, is built for the commissioning. A single WinCC OA project is dedicated for each subsystem and they are integrated as the MFT DCS via the network. Relevant components in terms of the FSM nodes are put together in a single DCS panel. The panels are refined to allow more user-friendly operations through the MFT surface commissioning stage.
Separate DCS panels are developed to be used by standard users and by experts. The DCS panels of a single RU control for standard users and experts are shown as examples in Figs. 15 and 16, respectively. Standard users are basically restricted to monitor the RU conditions. Only experts are allowed to turn on and off the power of the RU and to set the voltage values using the advanced setting functions. The FPGA configuration is easily accessible via a dedicated button. The temperature values of the RU board and chips on it are used to activate a safety interlock if any temperature value exceeds the threshold. Panels at the higher hierarchical stages can control all RUs simultaneously.

Summary and Outlook
The MFT is a new silicon pixel detector installed in the ALICE experiment for Runs 3 and 4 of the LHC at CERN, starting in 2021, in order to improve the muon tracking capability at forward rapidity. The MFT DCS has been developed within the frameworks of CERN and the ALICE DCS. It controls and monitors the low voltage power supplies, the detector and readout modules, and the cooling system of the MFT. The FSM is a key element both for operation of the detector and for the software interlock. The DSS is additionally used as the hardware interlock. The DCS is implemented, used, and tested in the quality assurance of the MFT ladders. It works perfectly at the smoke test of about 500 ladders. It is also integrated in the surface commissioning of the MFT, where its all functionalities are tested. The MFT DCS system is operational, constantly improving and adding needed functionalities, and ready for the real detector operation.
Interlock scenarios to be implemented on the FSM and/or DSS will be defined for specific alert cases. The DCS with full functionalities will be implemented to the ALICE detector and installed in the cavern for the physics runs.