New Fast Beam Conditions Monitoring (BCM1F) system for CMS

The CMS Beam Radiation Instrumentation and Luminosity (BRIL) project is composed of several systems providing the experiment protection from adverse beam conditions while also measuring the online luminosity and beam background. Although the readout bandwidth of the Fast Beam Conditions Monitoring system (BCM1F—one of the faster monitoring systems of the CMS BRIL), was sufficient for the initial LHC conditions, the foreseen enhancement of the beams parameters after the LHC Long Shutdown-1 (LS1) imposed the upgrade of the system. This paper presents the new BCM1F, which is designed to provide real-time fast diagnosis of beam conditions and instantaneous luminosity with readout able to resolve the 25 ns bunch structure.

The CMS Beam Radiation Instrumentation and Luminosity (BRIL) project is composed of several systems providing the experiment protection from adverse beam conditions while also measuring the online luminosity and beam background. Although the readout bandwidth of the Fast Beam Conditions Monitoring system (BCM1F -one of the faster monitoring systems of the CMS BRIL), was sufficient for the initial LHC conditions, the foreseen enhancement of the beams parameters after the LHC Long Shutdown-1 (LS1) imposed the upgrade of the system. This paper presents the new BCM1F, which is designed to provide real-time fast diagnosis of beam conditions and instantaneous luminosity with readout able to resolve the 25 ns bunch structure. K : Detector control systems (detector and experiment monitoring and slow-control systems, architecture, hardware, algorithms, databases); Diamond Detectors; Beam-line instrumentation (beam position and profile monitors; beam-intensity monitors; bunch length monitors); Particle identification methods 1Corresponding author.

Overview of the CMS Fast Beam Conditions Monitoring system
The increased performance of the LHC enabling collisions at 14 TeV, higher luminosity and 25 ns bunch spacing will increase the importance of real-time monitoring at high rates. The upgraded Fast Beam Conditions Monitoring (BCM1F) system is a part of the CMS Beam Radiation Instrumentation and Luminosity Project (BRIL). It is designed to monitor the flux and timing of the particles originating from the proton-proton interactions and machine induced background (MIB) particles. The frontend modules consist of 24 single crystalline CVD diamond sensors on two parallel planes positioned at a distance of z = ±1.83 m away from the interaction point (IP), mounted radially at r = 7.2 cm from the beam line. Each 5 mm × 5 mm × 500 µm diamond has a two pad metallisation with 25 µm split that will decrease the hit probability of a single channel. The distance between the sensors and the IP is optimal for the separation of incoming and outgoing particles and corresponds to the time-of-flight of 6.25 ns for relativistic particles. The hits originating from collisions, MIBs and activation will be classified using their arrival time. The corresponding count rates will be used to measure in real time the luminosity, background and activation product rates [1].

The BCM1F detector 2.1 Structure of the detector
The detector had been built using 24 single crystalline CVD 5 mm × 5 mm diamond sensors with a two pad metallization, each with a 25 um split. Diamond has almost five times wider band gap than silicon [2]. As a consequence it has a lower response to the ionizing particle hit, but a lower -1 -dark current even in the strongly irradiated areas. For this reason no cooling is required for the diamond detectors. In the BCM1F installation area where the space is very limited, this feature becomes advantageous. Another advantage of the diamond detectors is a high response speed due to the high charge carrier mobility in the diamond structure. Hence the detectors can be used for the particle arrival time measurement. The metalized diamonds are mounted to the front-end electronics (FEE). The FEE comprises a fast transimpedance preamplifier with active feedback, shaper stage and high-performance output buffer working as single ended-to-differential converter. The diamonds and the FEE are mounted on complex flex-rigid PCBs. These PCBs are held by a carbon-fibre carriage which is inserted in the pixel service tube. Each of four PCBs has a part dedicated to the analogue signals transmission to the back-end electronics. That is described in the following section in more details.

The new front-end electronics
A new FEE ASIC was designed in 130 nm technology. The project requirements for the design have been met. The FEE is optimised to operate with a detector capacitance in range 2 to 5 pF. The charge gain is in the order of 50 mV/fC depending on the FEE configuration. The pulses generated on the output have a peaking time of 6.6 to 9.6 ns depending on FEE configuration but not depending on the amplitude of the pulse. The pulse full-width-half maximum is within the range of 8.0 to 10.4 ns. The equivalent noise charge is in range of 300 to 750 e-for sensor capacitance in range of 2 and 5 pF. A feature which qualifies the structure for the high luminosity operation of the BCM1F is a fast baseline recovery after overdrive that is less than 25 ns for proper bias settings. Another important parameter is a high resolution of the overlapping pulses measured on the output. Pulses generated as the response of two consecutive hits at a time interval of 12.5 ns are fully distinguishable. Moreover, two MIB pulses following with a time interval of 10 ns can be still be resolved using a simple signal processing. The FEE has an output differential buffer that is able to drive a 100 Ω load. The linearity and gain measurement for various shaper bias values and 2 pF input capacitance are presented in figure 1. The good linearity is obtained for the input charges below 6 to 9 fC depending on the charge gain. Above those values the circuit starts to saturate until charge values around 15-20 fC. The structure has been tested in laboratory; the details of the tests are described in the report [3]. Photographs of the circuit mounted to the PCB with the square places for gluing the diamonds and the FEE mounted in the carrier are presented in figure 2.

Data transmission
Data from the detectors are transmitted as analog signals to the back-end electronics. 4 front-end PCBs have in total 48 detector channels that are distributed using a flexible PCB to four analogopto-hybrids (AOH) with 3-channels [4], mounted on a solid part of the PCB as presented in the picture in figure 2. Each AOH channel is equipped with a laser diode. The laser bias and gain are adjustable. For this reason, a single digital-opto-hybrid (DOH) [4] is installed. The configuration commands are formed and coded by the Slow Hub Controller that is implemented in the MicroTCA back-end electronics described in the following section. Transmission to the DOHs is performed digitally using SFP modules and the optical fibers. The DOH converts the signals to electrical, decodes them and provides the information to all four AOHs installed on the PCB. The address of -2 - Figure 1. Linearity of the front-end fast transimpedance preamplifier. The circuit has a good linearity for the input charges below 6 to 9 fC depending on the bias current. The charge gain varies between 33.1 to 75.9 mV/fC depending on a configuration [3].

Figure 2.
Pictures of the front-end electronics. On the left side: the front-end circuit is glued to the PCB, the square metalised place for the diamond sensor is visible; On the center: the front-end PCB with 6 ASICs mounted, the flex part of te PCB connecting the main part with the part designed for an optical transmission is visible; On the right side: two front-end PCBs with numbering of the sensors, the boards are mounted on two parallel planes positioned around the beam pipe.
the AOH is coded within the command so that only one module gets the configuration data and each laser can be tuned separately. From both sides of the detector the signals are transmitted to the counting room using optical fibers with length compensation between the near and the far side of the detector.
The end of the optical path is an optical receiver. Four CMS Opto Bahn 12 channel receivers of 100 MHz bandwidth are used. Output signals are provided in positive and negative polarity and are distributed to the back-end electronics. The negative signals are connected to the VME back-end electronics. The positive signals are provided to the MicroTCA back-end electronics for further processing. The back-end electronics systems is described in the next section.

The readout electronics
The back-end electronics is divided into two separate systems. The first system is a VME technologybased back-end that was upgraded on the basis of the experience collected in the previously working system. The system operates continuously, for all operational modes of the LHC, independent of the CMS data-taking status. It is compatible with the new front-end electronics and has relatively low resolution in time that is 4 bins per 25 ns bunch crossing. Its double pulse resolution is 7 ns. The following detailed description of the system presents the architecture and each element separately. The second system, consists of a newly designed MicroTCA based back-end electronics and introduces a novel signal processing algorithm. The new technology allows advanced signal processing which is implemented in the FPGA. The dead-time free operation makes it more convenient for high luminosity operation. It is also fully compatible with the front-end and additionally it resolves the double hit front-end signals. With a time resolution of 31 bins per 25 ns bunch-crossing it is able to distinguish collisions from the machine induced background particle hits. This system is still being developed and a first complete release is foreseen by the end 2015. The MicroTCA based system is presented in section 4.2.

The VME based back-end electronics
The negative signals from the optical receivers are transmitted using 4 ns cables to the fan-out that duplicates the signal. One set of outputs is connected to analog-to-digital converters CAEN ADC V1721 8 channel digitizer modules. They are equipped with 8 bit analog-to-digital converters (ADC) working at a 500 MS/s sampling rate. The ADCs are able to collect up to 45 beam orbits in a ring buffer with time stamps. It can be triggered externally or internally. The module is used for the present efficiency and signal characteristics monitoring. Data produced are read through an optical link and sent to a PC for processing. An example data set collected during 50 × 10 3 orbits is presented in a histogram, figure 3. The x-axis is the signal amplitude in mV with a 4 mV resolution that corresponds to the most probable amplitude of the detector response to a particle hit. The y-axis is the number of particles for a given amplitude. The maximum of the histogram indicates the most probable amplitude of the detector response to a minimum ionizing particle hit. The signal cut is set at 20 mV.
The second set of signals from the fan-out is connected to CAEN V895 discriminators. Each module has 16 input channels, so 3 boards are needed to process all 48 detector channels. Each channel has a threshold individually programmed. The discriminated signals are sent to a CAEN V1495 logic unit, working as a look up table (LUT). The unit comprises (in its actual configuration) 48 digital detector inputs (ECL), 10 clock and auxiliary inputs (NIM) and 72 digital outputs (LVDS, ECL and NIM). The signal processing is performed in Altera's CYCLONE II FPGA. The module operates with a 320 MHz internal clock which is phase synchronized to the LHC clock. The firmware manages the internal processing as well as mapping of signals to the I/O channels. The registers are accessible through the VME interface. The default settings of all vital parameters can be programmed into firmware. The LUT also provides a variety of logic triggers based on the bunch detector signals BPTX of the CMS detector which are sent to the central trigger.
The signals are further processed by a real-time histogramming unit (RHU) designed in DESY Zeuthen [5]. The board is designed for sampling and processing discriminated signals in real time -4 - Figure 3. An example of the BCM1F amplitude spectrum. Signal amplitude cut A > 20 mV. The VME ADC data was collected with colliding beams in the LHC. The signal amplitude spectrum was built using a simple peak finding algorithm. The main peak around 70 mV corresponds to 1 MIP and that around 140 mV to 2 MIPs. and sending via a network. It measures the arrival time of hits relative to the orbit trigger of the LHC. The RHU incorporates an FPGA with 5 Mbit internal memory, 16 Mbit external memory and a single board computer (SBC). The FPGA samples the detector channels with a frequency of 160 MHz, synchronously to the LHC bunch clock. It produces real time histograms collecting the particle hits as a function of time. The hits are processed over 2 12 orbits and mapped in the time of one orbit. Each histogram has a 6.25 ns binning and contains 14256 bins. The external memory is used to store the previous 50 orbits of sampling data in a ring buffer. Thus the data can be used for a postmortem analysis in case of a beam dump. The histogram buffers are read in packets of 2048 using the Linux based SBC and transferred to the BRIL DAQ. For a dead time free operation, double buffering is implemented. The SBC readout bandwidth is 20 MB/s. The SBC incorporates a 32Bit ARM926 Core at 400 MHz and 128 MB of RAM. A Linux Kernel driver has been implemented to achieve full performance at the interrupt-based read out. In the user space, a TCP server waits for connections and sends the data to connected clients. A software framework has been developed to allow access to the data of the RHU via network and can be used with ROOT programs for data analysis. The framework uses shared memory to allow multiple clients on the same machine to reuse the network connection to the RHU device and to save bandwidth.
The part of the VME based system used for monitoring, suffers from a considerable dead time introduced by the CAEN ADC boards. A new architecture, which is presented in the following section, is able to operate in real-time as well as providing advanced signal processing capabilities and overlapping pulse resolving.

The MicroTCA based back-end electronics
The architecture of the upgraded BCM1F back-end electronics is designed in the MicroTCA technology [6]. The system control and communication is operated by the specialized CMS MicroTCA Hub module AMC13 (AMC13 XG) and the NAT MicroTCA Hub module (MCH) [7]. A communi--5 -cation protocol for the system is IPbus, an IP based protocol that is developed by a UK collaboration for controlling the CMS trigger and readout [8]. The system is presented in figure 4. The system uses 2 MicroTCA crates. Each crate contains 12 GLIB AMC carriers with High-pin count (HPC) FMC connectors mounted. The HPC connector is required for using 4 ADC channels of FMC125 mezzanines at the maximum rate. The FMC125 modules are commercial products developed by the 4DSP company [9] and customized for the project requirements. The modules are synchronized using the LHC clock that is provided to the mezzanines through the front panel. The 8-bit samples are transmitted through the FMC connectors to the XILINX Virtex-6 FPGAs. The timing information and control commands are received by the AMC13XG module from the Trigger, Control and Distribution System (TCDS) [10] for CMS. The signals are transmitted through the backplanes to all GLIB modules where they are decoded and used for synchronization with the CMS commands. The processing FPGA uses this information for synchronization and incoming data. One orbit (89.1 µs) of a raw data for each 2 14 orbits is stored in an internal FIFO memory. This data set is requested and read out by the software when the buffer is ready. There is a single buffer implemented, which is sufficient, as the data is stored as a reference. After an offline processing, the data is used as a reference for the histograms produced by firmware. The histograms are stored in parallel to the raw data collecting as described below.
The amplitude monitor of the detector gain, produces an amplitude spectrum of the detected particles corresponding to the energy loss distribution of relativistic charged particles traversing the -6 -

JINST 11 C01088
sensors. An example of the histogram collected using the MicroTCA FMC during the first runs is presented in figure 4. The measured values correspond to the front-end electronics response to a particle hit. The amplitude resolution of the histogram is 256 bins. In the plot, the bins are merged in groups of three due to the low statistics collected. The amplitude histogram is filled during the LHC orbit and integrated over 2 14 orbits, which allows continuous monitoring of the performance of each channel. The calibration pulses for gain monitoring are generated by the front-end ASIC calibration circuitry. The time domain histogram will provide information about the distribution of the hits in the LHC orbit. It will reflect the LHC bunch time structure observed on the basis of the incoming hits. A histogram will be filled during a 2 14 orbits time interval and its aim is to provide the most reliable information about the collision particles and background. In comparison to the histogram produced by the RHU module, the resolution in time will be 31 bins / 25 ns. The high sampling frequency allows the implementation of the signal processing for the high accuracy signal parameters measurement as well as the detection and measurement of the overlapping pulses.
The histograms are stored in the internal memory and transmitted to the MCH through the backplane and subsequently to the BRIL DAQ. In order to operate without introducing a dead time, double buffering is implemented. The data will be published in the format it is collected, without processing in software and saved as a single histogram collected during a run. The data for storage will be collected by the software. Use of the AMC13 and the S-Link connection for the data transmission and the MCH only for the commands transmission and the diagnostics is considered for a final system architecture. The details concerning the data storage and transmission rates are described in the next section.
-7 -A separate branch of the processing applies to signals from the BPTX detector. The BPTX detector uses two standard LHC beam position monitors (BPM) each comprising of four electrostatic button electrodes positioned symmetrically around the beam-pipe. Comparison of timings from opposite beam position monitors gives the highly accurate measurements of the longitudinal interaction point and bunch timing relative to the CMS clock. For this purpose, 2 GLIB boards with FMC125 modules are installed in the upper crate. The mezzanines are configured in a 1-channel working mode with the maximum 5 Gs/s of the sampling rate.
The last important element of the system will be the readout monitoring. For this purpose an additional GLIB board is installed with a custom FMC mezzanine EDA-02707-V1 that hosts eight sockets for SFP optical modules operating at up to 800 Mbit/s. The module provides signals for control and configuration of the Linear Laser Drivers (LLD) [11] of the AOHs and DOH being the part of the front-end. The monitoring system is based on the CMS CACTUS environment which provides an IPbus communication between the software and FPGA. The application allows for configuration of the bias current and gain setting of the lasers. The lasers operate in a high radiation field, so the parameters will change with the accumulated radiation dose and corrections will be required. The firmware driver communicates with the AOHs/DOH through the optical links. For this reason the FMC mezzanine EDA-02707-V1 housing 8 optical COTS (Commercial of the Shelves) SFP transceivers is mounted on the GLIB AMC. The optical signals are converted to electrical signals by the DOH and then sent to the Slow Hub ASIC [12]. The chip converts the coded data to the I2C commands for LLD, activates one of the 7 possible output channels and sends the configuration. The hub has been designed for the Forward Pixel Detector.

Data readout software
Data stream is built based on a specialized network and software suite to provide low latency and high availability of the readout system. The network includes a software concentrator and request broker called Control Hub which facilitates the communication with IPBUS Devices. The control hub is written in erlang/OTP and designed for highly efficient real-time communication with the distributed systems. The acquisition software implements a BRILDAQ generic model of the source/processor architecture. Raw data acquisition with the hardware configuration/monitoring is separated from the data processing and normalization. Finally produced physics data are distributed across CMS and saved in a persistent storage.

Summary and outlook
The BCM1F system is operating successfully until spring 2015. It operates continuously as a simple state device and provides the real-time beam quality measurement. It provides the delivered and recorded luminosity measurement as well as the machine-induced-background rates. It was calibrated using the Van der Meer scans technique. The background measurement provided by the system is used by tracker and pixels for automatic switch-on semaphore. Data is collected, processed and stored using the BRILDAQ. The increasing luminosity challenges the system with an increasing pile-up and increasing number of the activation products. The further upgrades of the system are targeted into providing the online luminosity measurement that is linear with pileup. The -8 -

JINST 11 C01088
detectors need to provide the fast response to the single and overlapping events without introducing a dead time. The novel methods of the luminosity measurement are being developed using the previous and currently collected experiences.