Measurements of Single Event Upset in ATLAS IBL

Effects of Single Event Upsets (SEU) and Single Event Transients (SET) are studied in the FE-I4B chip of the innermost layer of the ATLAS pixel system. SEU/SET affect the FE-I4B Global Registers as well as the settings for the individual pixels, causing, among other things, occupancy losses, drops in the low voltage currents, noisy pixels, and silent pixels. Quantitative data analysis and simulations indicate that SET dominate over SEU on the load line of the memory. Operational issues and mitigation techniques are presented.


Introduction
During operation, electronics at the Large Hadron Collider (LHC) experiments must contend with radiation damage both from integrated dose and from instantaneous Single Event Effects (SEE) [1]. The most prevalent SEE is the flipping of a logic state, typically in a memory cell, known as Single Event Upset (SEU). The readout integrated circuits in the ATLAS pixel detector were custom designed with SEU-hardened memory cells [2] (Dual Interlocked CElls or DICE latches [3] and triple redundancy). These reduce the SEU rate, but do not completely eliminate it. Another frequent -1 - ation, the performance of the innermost el detector, the so-called b-layer, will start years of LHC working. So, it is proposed b-layer detector. For this purpose, electronic design of the pixel front end is elopment using the 130 nm process [4]. the b-layer replacement, the level of imes higher than at the start of the LHC. stimated to reach 150 to 200 MRad and lose to 1x10 8 particles/cm²/sec. e commercial 130 nm process used to chip is less sensitive than older process effect of the total ionizing dose and ade on individual devices are very ve to consider carefully the SEU for this s. In fact, the device dimensions are small of storage nodes becomes lower. The ed is low (1.0 V to 1.4 V for the 130 nm al charge needed to provoke an upset n in older processes and digital designs against SEU. flops are not suitable to be used in the nt. D flip flops based on the dual CE) latches have redundant storage nodes ll original state when an SEU error is le node [5]. The probability that multiple y an upset is low, making the DICE latch . However, as the device size shrinks, the

A. DICE Structure
The DICE latch structure is shown in Figure 1. It is based on the conventional cross coupled inverter latch structure. The 4 nodes X1 to X4 store data as 2 pairs of complementary values.   27. A majority logic voting circuit is implemented in each cell to determine the effective output data. This the data bit is corrupted only when more than one latch is upset. During the write cycle, the global WE signal is activated and the selection line L i corresponding to the bit i is activated 1060 through the address word. In this case the load signal (load i ) is activated. Each DICE cell has its own load signal to avoid corrupting the three latches if a transient upset occurs in the preceding combinatorial logic. The DICE latch structure is shown in Fig. 28 and consists of cross-coupled inverters. The 4 nodes (n 1 to n 4 ) store data as 2 pairs of complementary values. For example, when the stored bit is 0 then n 1 ,n 2 ,n 3 ,n 4 = 0101. If we assume a positive upset pulse on the node n 1 , 1065 the transistor MP2 is blocked avoiding the propagation of this perturbation to the node n 2 . At the same time the transistor MN4 will propagate a negative pulse to the node n 4 blocking MN3 and avoiding n 3 level corruption. The perturbation is then removed after the upset transient since the SEE is the creation of short pulses, called Single Event Transient (SET) or glitches. These glitches may propagate through the chip and flip the logic state of the memory. A sufficiently large amount of charge must be injected into the sensitive nodes of a circuit to flip its logic states. This corresponds to an ionization threshold. Even for memory cells that are not hardened, this threshold is much higher than the specific ionization of minimum ionizing particles (MIPs). Thus, at the LHC, SEUs are not caused by the MIPs that the detectors were built to measure, but by highly ionizing recoil nuclei and by showers resulting from occasional nuclear interactions of the MIPs in the proximity of the memory cells [4]. These secondary interactions are complex to model, and it is therefore interesting to measure SEU rates in-situ and compare these rates to test beam results acquired during the detector construction phase. It is also important to quantify the operational consequences of SEUs in FE-I4B.
In this paper we analyze data from operation of the ATLAS innermost pixel layer, called the Insertable B-Layer (IBL) [5]- [6]. We quantify the influence of SEU on the pixel matrix and on the data readout, and extract SEU cross sections that can be compared to test beam data. The results have led to improvements in operation of the present detector and mitigation of SEU effects, and they provide an important reference for the design of future detectors.
Single DICE latch memory cells (see figure 1) are used to store pixel configuration values, while triple redundant DICE latches (see figure 2) are used to store chip-wide (or global) configuration values.
The layout at transistor level is optimized to maximize distance between coupled nodes and between redundant latches, for maximal reduction of upset rates. Upset of the specific pixel configuration latch can usually only affect a single pixel at a time, while upset of global configuration memory can impact many pixels and even the operation of the entire chip. It is possible to directly detect upsets in the global configuration, but not upsets in the local pixel configuration. While it is possible to read back pixel latch values (see section 5), this requires special operations not typically    . A majority logic voting circuit is implemented in each cell to determine the effective output data. This the data bit is corrupted only when more than one latch is upset. During the write cycle, the global WE signal is activated and the selection line L i corresponding to the bit i is activated 1060 through the address word. In this case the load signal (load i ) is activated. Each DICE cell has its own load signal to avoid corrupting the three latches if a transient upset occurs in the preceding combinatorial logic. The DICE latch structure is shown in Fig. 28 and consists of cross-coupled inverters. The 4 nodes (n 1 to n 4 ) store data as 2 pairs of complementary values. For example, when the stored bit is 0 then n 1 ,n 2 ,n 3 ,n 4 = 0101. If we assume a positive upset pulse on the node n 1 , 1065 the transistor MP2 is blocked avoiding the propagation of this perturbation to the node n 2 . At the same time the transistor MN4 will propagate a negative pulse to the node n 4 blocking MN3 and avoiding n 3 level corruption. The perturbation is then removed after the upset transient since the v2.3 48 load in Figure 28: Single DICE latch structure.
nodes n 2 and n 3 have conserved the right information. However, if two sensitive nodes of the cell storing the same logic state (n 1 and n 3 ) or (n 2 and n 4 ) change state due to the effect of a single 1070 particle impact, the immunity is lost and the DICE latch can be upset. For this reason, special layout rules are followed to physically separate sensitive nodes as shown in Fig. 29. An enclosed layout nMOS transistor is used to achieve a wide gate with reduced diffusion area, which reduces the upset probability (the charge deposited by an ionizing particle is collected by the diffusion and translated to a voltage by the gate capacitance).
1075 Figure 29: Special layout used to enhance tolerance of DICE latches.

EFUSE
The EFUSE block consists of a bank of triple redundant RAM of the same design as the CNFG-v2.3 49 19 / 21 Figure 2. The FE-I4B global memory cell is based on the triplication of DICE latches with the addition of the simple majority logic. No automatic error correction is used, but single bit flips can be monitored by an error counter. The memory has two independent ports. The first (Datain, Load) is used to write data coming from the command decoder and the second (To-config, To-readout) to read the majority logic output. The output data bit is corrupted only when more than one DICE latch is upset.
performed during data taking. Different techniques have been applied to measure pixel latch updates in physics data without directly reading out the stored values. These involve understanding the effect of flipping each configuration bit and searching for these effects in data. The main effects studied are called quiet pixels (when a pixel's efficiency drops abruptly, see section 4.2.1) and noisy pixels (when a pixel suddenly fires at a high rate, see section 4.2.5). In special runs, pixels can be deliberately configured to begin in the off state, and a search is performed with the aim of monitoring when they become active, see section 4.2.3.
During 2017, the LHC peak instantaneous luminosity increased with respect to 2016 and was constantly above 1.5 × 10 34 cm −2 s −1 . Under these conditions, the effect of SEUs on the FE-I4B global registers was greater than in previous years, and more frequent front end chip reconfigurations were needed to preserve good data quality and data taking efficiency. The effects of SEUs were visible in the behavior of the FEs: impacted FEs can stop sending hits, become very noisy, or experience large drops/increases (up to ±100 mA) of the low voltage (LV) current consumption monitored from the Detector Control System (DCS) (see figure 3). Starting in 2016, some SEU/SET effects were addressed by manual or automatic ("Quick Status") reconfiguration of the affected modules.
To minimize the impact of SEUs on ATLAS data taking, a procedure was implemented to regularly reconfigure the global registers of the FE-I4B readout chips in the entire IBL. Thanks to a joint effort of online software and firmware (see section 6.1), it was possible to introduce this procedure without any additional dead time in ATLAS. Starting in August 2017, the global registers of the IBL FE-I4B chips were reconfigured approximately every five seconds, improving the overall Data Acquisition (DAQ) efficiency and eliminating the low voltage current fluctuations that were previously observed. Unfortunately it was not possible to also regularly reconfigure the single pixel DICE latches in the FE-I4B: the software modifications needed impacted the overall stability of the   . Effects of SEU on FE-I4B global registers can be seen during a typical LHC fill; in this case (fill 6046), the peak luminosity reached is 1.5 × 10 34 cm −2 s −1 , and 490 pb −1 were delivered over the entire fill. During the data taking, at luminosity block 1 (LB) ∼ 268, a drop in the LV current consumption can be observed (green line). At the same time, a drop in occupancy is observed in one (pink line) of the two DAQ modules that share the same LV power supply (see the Appendix for details about the DAQ and power system granularity). At LB ∼ 277, the DAQ module was manually reconfigured, bringing the LV current consumption and hit occupancy back to their values before the SEU. DAQ system. However, a test run was performed in July 2018 and can be used as proof of concept for future implementations (see details in section 6.2).

The ATLAS IBL detector
The original ATLAS Run 1 Pixel detector was upgraded for Run 2 with the addition of the IBL, which provides an additional layer of pixels at ∼ 3.4 cm from the interaction point. The IBL greatly improves impact parameter resolution, vertexing, b-tagging and tracking robustness after high radiation doses. The IBL detector consists of one layer of pixel sensors (see figure 4) built around the beam pipe, providing full azimuthal (φ) hermeticity for high transverse momentum (p T > 1 GeV) particles and longitudinal coverage up to |η| = 3. Two different sensor types were chosen: a planar design in the central area (|η| < 2.5) based on the already existing ATLAS Pixel sensor, and 3D sensor designs in the forward regions (2.5 < |η| < 3) which are applied here for the first time in a high energy physics experiment.
The IBL uses the FE-I4B readout chip [7], [8], which is built in 130 nm CMOS technology. A top level diagram of the FE-I4B is shown in figure 5. Pixel sensor with each pixel nominal size of 1Luminosity Block (LB) is a time interval of ATLAS data recording over which the experimental conditions are assumed to be constant. LB duration is usually around one minute.  250 × 50 µm 2 is bonded to FE-I4B chip. On the edges of the chip there are long pixels with the size of 500 × 50 µm 2 or 450 × 50 µm 2 [6].
The detector is made of 14 carbon fiber staves, 64.3 cm long and 2 cm wide. A transverse view of the IBL staves is shown in figure 6. The Stave ID numbering scheme counts from 1 to 14, increasing with the φ angle.
-5 - Each stave hosts 20 pixel sensor modules (see figure 7) corresponding to 12 double-FE-I4B chip modules (for the planar sensors) and 8 single-FE-I4B chip modules (for the 3D sensors). In total, there are 32 FE-I4B chips per stave, equivalent to 448 FE-I4Bs and ca. 12 million pixels to be read out for the entire IBL.
The IBL readout electronics are designed to exploit the enhanced detector performance and overcome the limitations present in ATLAS Pixel Run 1 system. The IBL readout system consists of two 9U-VME cards: the Back-of-Crate (BOC) and Read-Out Driver (ROD) which implement optical I/O interface and data processing, respectively (see figure 8). Each card pair processes data received from a full IBL stave, corresponding to 32 FE-I4B data links working at 160 Mb/s, for a total I/O bandwidth of 5.12 Gb/s. Moreover, the new ROD design implements dedicated Gigabit Ethernet connections, thus enhancing the overall acquisition performance of the calibration runs. Configuration and control of the read-out electronics are the main tasks of the ROD card. In order to have new capabilities, a PowerPC (PPC) was embedded into the main ROD FPGA, a Xilinx Virtex-5 (see figure 8). The PPC solution enhanced the control and monitoring performance; in particular, a specific software task to recover the FE-I4B from SEU was developed and deployed in Further details about the IBL detector layout and naming convention are covered in the Appendix, including the naming schemes used for the power system, the data-acquisition modules, the sensors, and the front-end chips.

Global and pixel configuration memories
A 13 bit register is available to configure each pixel: it is comprised of a 1-bit enable flag, a 5-bit threshold tuning DAC (TDAC), a 4-bit time-over-threshold (ToT) tuning DAC (FDAC), a 1-bit HitBus (input to logical OR of all pixel discriminator outputs in the matrix), and 2 bits for the selection of the charge injection capacitor. The ToT represents the length of time that single pixel discriminator is over threshold. It has a 4-bit resolution, in counts of an externally supplied clock, nominally 40 MHz, that corresponds to the LHC bunch crossing (BC) time of 25 ns.
The pixel analog front end of FE-I4B is shown schematically in figure 9. Single DICE latches are used to store bits within each pixel.
In data taking configuration, the output enable bit mostly stores ones, as most of the pixels are enabled. However, a small fraction of pixels (tenths of a percent) corresponding to noisy pixels are disabled during calibration. The TDAC value is centered around fifteen, and the FDAC value is typically centered around seven. The capacitor selection bits are both set to one, and the HitBus flag is set to one (meaning HitOr disabled).
The Global Configuration Memory, located at the end of the column area outside of the pixel matrix region, is implemented as a memory block of 32 words of 16 bits (512 bits in total). The design used for this global memory is based on the triplication of the DICE latches to further suppress SEU. Such triplication is not possible inside the pixel due to space constraints. The global threshold, generated by a coarse and a fine DAC, is vital for proper chip functionality. Together the two DACs give a unique Global DAC (GDAC) value that is typically tuned, along with local TDAC registers, to achieve a target value of 2500 e − threshold for each pixel.
The occurrence of SEUs during data taking modifies both single pixel and global configurations, producing quiet pixels, noisy pixels, and introducing a general detuning of the FE-I4.

SEU measurements
The SEU cross section is equal to the total number of bit flips observed in the whole memory (N flips ) over the proton fluence achieved during the test (Φ), and divided by the number of latches implemented in the chip (N latches ):

Beam tests
Irradiation tests were carried out in 2012 using the IRRAD3 beam line of the Proton Synchrotron (PS) Irradiation Facility [9] at CERN. The IRRAD3 provides a beam of 24 GeV protons, and the structure of the beam is defined by the operation cycle of the PS accelerator.
-8 -During the irradiation testing, devices were oriented perpendicular to the radiation beam line. The enable bits of all pixels were disabled before the test beam. The 0 → 1 transition of the pixel enable bit causes a pixel to respond to charge injection. The test beam was periodically injected and the number of active pixels was measured. The SEU cross section measured in the 2012 beam test was 1.1 × 10 −15 cm 2 [2] for a DICE latch implementation (FE-I4A, columns 60-75) that closely matches the implementation in FE-I4B.

Analysis of cluster data from physics running
SEU effects can flip the enable bit of a pixel from 1 to 0, making the pixel quiet. The SEU flip of a TDAC bit can lower the pixel threshold and make the pixel noisy. Therefore, the sensitivity of DICE latches to SEU can be estimated by studying such quiet or noisy pixels.  Figure 10 shows the occupancy of each pixel in one 3D module in the most forward IBL ring (with a width of one FE-I4B chip) on the negative-z side. The data are extracted from randomly triggered events in the first 16 pb −1 of LHC fill 7018 in 2018. Long pixels, which have more hits than normal pixels, are not included in the sample. The average occupancy of one pixel in 16 pb −1 of data from this fill is about 155.

Quiet pixels
A pixel is defined to be quiet if it fired zero times in 16 pb −1 of data taking. Figure 11 shows the fraction of quiet pixels versus integrated luminosity (L) in the 14 modules in the most forward IBL ring on the negative-z side. The number of quiet pixels increases during data taking due to the accumulation of pixels with a flipped enable bit. In two modules one can observe a fast drop in the -9 - number of quiet pixels due to the reconfiguration of these modules. Module-to-module differences in the initial number of quiet pixels indicate different fractions of silent pixels, which fire zero times during the entire fill. The fraction of pixels that become quiet due to SEU is seen to increase linearly with integrated luminosity. The dependence of the number of quiet pixels on luminosity was fitted with a linear function p 0 + p 1 · L, where the mean p 1 is (5.4 ± 1.3) × 10 −3 fb. The fraction of pixels that become quiet due to SEU (p 1 · L) is equal to the ratio N errors N latches in Eq. 4.1. The pixel latch SEU cross section in FE-I4B is calculated with the "quiet-pixels-fraction": The predicted flux of hadrons with energy above 20 MeV with PYTHIA/FLUKA [10]- [11] simulations in the extreme outside 3D sensor IBL module is Φ = 0.91 × 10 13 hadrons (energy above 20 MeV) cm −2 per 1 fb −1 [12]. The SEU cross section is calculated to be (0.60 ± 0.14) × 10 −15 cm 2 , which is of the same order of magnitude as the test beam result. Figure 12 shows the average fraction of quiet pixels in each chip ring after ∼480 pb −1 of data taking in LHC fill 7018. It is compared with the PYTHIA/FLUKA simulation, which is normalized to the average fraction in data.

Z-dependence of appearance of quiet pixels
-10 - The numbering of the FE-I4B chips on 3D modules is in Ref. [13] and in the Appendix.

Measurement of the change of enable bit fraction during an LHC fill
A measurement of the change in the enable bit was carried out by disabling the output enable bits of all pixels in four FE-I4B chips at the beginning of LHC fill 6356. The pixels firing at the passage -11 -of a charged particle indicate SEU-induced transitions of output enable bits. Figure 13 shows the fraction of pixels, initially disabled at the beginning of the LHC fill, that are enabled by SEU. The rate of increase in enable bit 0 → 1 transitions obtained during this LHC fill is consistent with the increase rate of the quiet pixels in figure 11.

Broken primary clusters
The number of clusters produced by primary charged particles from a 13 TeV pp collision vertex (referred to as "primary clusters") is proportional to luminosity [14]. Quiet pixels can cause clusters to be broken. In the 3D modules, the average length of a primary cluster is nine pixels, so broken clusters are a very sensitive probe of quiet pixels caused by a flip of the SEU enable bit from 1 → 0. Figure 14(a) shows the center-to-center distance along the azimuthal direction (∆row) for all pairs of clusters separated by a one-pixel gap along the z direction. The data shown is from the most forward 3D module on the negative-z side, taken during the first Luminosity Block (LB) of fill 5163 in 2016. If two clusters are separated by a one-pixel gap along z and ∆row is no larger than three pixels, then these clusters are assumed to be broken from one long cluster. Background where two clusters are close together due to random coincidence is flat in ∆row. This background level depends on the cluster density. By requiring that the longitudinal length of the merged cluster be consistent with a primary particle, broken background clusters from sources other than primary particles can be excluded from the ∆row distribution [14]. The number of broken primary clusters in each LB is obtained by fitting the ∆row distribution with a double-Gaussian describing the signal and a first-order polynomial describing the background. The total number of primary clusters in each LB is obtained by fitting the longitudinal cluster length distribution in Ref. [14]. Figure 14(b) shows the fraction of broken primary clusters versus integrated luminosity in fill 5163 from 2016, in the 8 rings of 3D modules. The 14 modules in the same ring have been studied together to achieve higher statistics. Module-to-module differences in the initial number of broken clusters indicate different fractions of silent pixels in the different rings. All of the rings show a similar increase in the number of broken clusters with luminosity.

Noisy pixels
Noisy pixels are studied with the same LHC fill as above, and the analysis strategy is similar to that used to study quiet pixels. A pixel is defined to be noisy if it fired more than 350 times during the 16 pb −1 slice of data taking. Figure 15 shows the fraction of noisy pixels versus integrated luminosity in the 14 modules in the most forward IBL ring on the negative-z side. We assume the pixels become noisy due to a threshold decrease. In two modules one can observe a fast drop in the number of noisy pixels due to the reconfiguration of these modules. How SEU effects make pixels noisy is explored in section 4.3.

Noise due to TDAC MSB bit flip
The threshold in each pixel is determined by:  configuration. The pixels with more than 200 hits in this fill are defined as noisy. Low values of TDAC correspond to high thresholds. The increase in the fraction of noisy pixels with initial TDAC < 15 indicates that some pixels become noisy due to the SEU flip 0 → 1 of the most significant bit (MSB) of TDAC, which lowers the pixel threshold by ∼1850 e − . No correlation of the noise with FDAC values was observed.

Manual reproduction of the TDAC bit flip
To mimic an SEU on the MSB of TDAC, the MSB is set to 1 in the laboratory for all pixels on an individual FE-I4B test chip, which has been tuned with a target threshold of 2500 e − . The chip is bonded to a non-irradiated sensor. TdacVbp (step size of the least significant bit of the in-pixel threshold tuning DAC) is set to 100. As shown in figure 17(a), the TDAC distribution is shifted when TDAC MSB is set to 1 for all pixels. Figure 17(b) shows the threshold distributions after the MSB of TDAC in each pixel is set to 1 and the module was retuned with TdacVbp=125. Thresholds below 500 e − cannot be measured (due to a lower limit of the charge injection) and are defined to be 0. The threshold of some pixels decreases due to a flip of TDAC-MSB.
The average noise occupancy distribution in figure 17 (c) is obtained with a noise scan 3 by sending 3 × 10 7 triggers. Many pixels become noisy due to the threshold decrease caused by the 0 → 1 transition of TDAC-MSB. The FE-I4B module is more robust against SEUs when TdacVbp is smaller, but some pixels cannot be tuned at low TdacVbp. The fraction of noisy pixels is different among different chips. Could these differences be explained due to the different TdacVbp values? In fact 99% of FE-I4B chips in the IBL have TdacVbp=180. In particular, 13 out of 14 chips included in figure 15 have TdacVbp=180 (the module denoted by black circles has TdacVbp=140). Therefore, differences in the fraction of noisy pixels cannot be explained by different values of TdacVbp. These differences are partly due to the differences in the mean TDAC values in the chip and probably also related to different offsets of the effective global thresholds between the modules or to differences in the global noise levels.

L-shaped noise map in 3D sensor modules
One consequence of SEE effects in FE-I4B was the observation of an L-shaped noise structure in reworked (∼50% of all) FBK-type 4 3D sensor IBL modules (See figure 18). On these 3D modules, the output of the HitOr signal was accidentally wire bonded to the flex cable attached to the back side of the sensor. On the FBK sensors, the metallization on the back side is not continuous, but has a grid-like structure, which allows some cross talk from the HitOr line on the flex cable into the 3D sensor. Normally the HitBus outputs are disabled, so during laboratory and low luminosity 4Modules with 3D sensors produced at FBK factory [5].
-15 - runs this cross talk was not observed. But during high luminosity runs, several HitBus bit flips are produced by SEE effects, and the signals on the HitOr flex cable line induce cross talk signals in the pixels under this line with this particular L-shape. This effect has no visible consequence for physics, because the 3D modules are outside the tracking acceptance of |η| < 2.5, but it provides several lessons for future detectors: to use full backside metallization in the sensor, to avoid close contact of flex cables with the sensors, and to remove unused wire bonds between the chip and the cables.

Read-back measurements
The FE-I4B provides the functionality to read back the content of the DICE latches by copying the latch content in the Shift Register (SR) for each double column and transmitting it back to the Readout Drivers (RODs). The SEU measurement scheme is to write to a register in the device being tested, let a certain fluence of particles traverse the device, and then read back the register.

Single pixel configuration read-back measurements
The fraction of pixels in which the bit state flips ( N i (0)−N i N i (i = 0, 1)) after taking data of integrated luminosity L depends on the probability of 0 → 1 (P 0→1 ) and 1 → 0 (P 1→0 ) transitions: In figure 19, the status of TDAC-bit2 in each pixel is checked at the beginning of LHC fill 6371 and after about 150 pb −1 of data for 3D sensors and after the end of LHC fill for planar modules, -16 - respectively. Read-back cannot be performed while a FE receives trigger signals, so there are only two read-backs for planar modules: before the start of the collisions and after beam dump. The intermediate point is from 3D modules. The read-back for 3D modules were done more frequently, because 3D modules are outside tracking acceptance of |η|<2.5, so for small number of runs we could remove 3D information. The first point is zero by construction. P 0→1 and P 1→0 are extracted from these measurements. This procedure allows for an independent measurement of the SEU probabilities for each of the 13 pixel latches.
The 0 → 1 and 1 → 0 transition probabilities have been measured for the pixel latch memories. Only the results of the latches most important for DAQ operations are reported. In figure 20 (a), the P 0→1 of output enable bit has not been calculated because most of the pixels are enabled initially. In figure 20 (b) and (c), the 0 → 1 transition appears to be more frequent by a factor of ∼ 5 with respect to the 1 → 0 transition for the FEs investigated. The simulation of the SEUs in DICE memories [15], [16], [17] demonstrates that the dominant effect comes from glitches on the LOAD signal. The LOAD signal stores into the DICE latch the current content of the SR. In this case, the polarity of the transition depends on the actual values stored in the SR. The last bit (out of 13 bits) loaded into the SR is the output enable bit. As the enable bit is usually 1, this favors 0 → 1 transitions.
These asymmetries were studied in detail during LHC fill 7334 with the read-back method. The probability of a pixel memory bit flip strongly depends on the value stored in the input Shift Register. Pixel memory read-back results and chip simulations confirm that the FE-I4B SEEs are dominated by Single Event Transient effects (SET), which create fake LOAD signals. 0 → 1 flips in pixel memory are dominant when the SR is loaded with 1, and 1 → 0 flips are dominant when SR is loaded with 0. If the glitches on the LOAD line are created in the transistors close to the memory cell, only one bit is reloaded, but if the glitches are created in the transistors upstream, several bits could be reloaded simultaneously. The average rate of the SEU/SET bit flips in pixel memory of FE-I4B per fb −1 was studied in LHC fill 7334 as a function of the chip number. In figure 21 (a), the Shift Register (SR) was set to 1, and 0 → 1 flips dominate due to the SET on the LOAD line, while low rate 1 → 0 flips are due to real memory SEU. In figure 21 (b), the Shift Register was set   Simulations of SET effects in DICE memories [15], [16], [17] demonstrate that most of the SET on the LOAD line are produced by heavy ionizing particles simultaneously crossing the pair of transistors T1 and T2 (or alternatively T3 and T4). Unfortunately, in the current implementation of the DICE latch (see figure 1) in FE-I4B, the transistor T1 is close to T2, and transistor T3 is close to T4. Correlated glitches created by the same particle in both transistors are therefore relatively probable. In the same simulations, it was shown that one can considerably reduce these glitches by increasing the distance between the transistors T1-T2 and T3-T4.
In the same fill, the average rate of the SEU/SET bit flips in pixel memory was studied as a function of the bit number (0-12). In figure 22 (a), the Shift Register was set to 1, and 0 → 1 flips dominate due to the SET on the LOAD line, while low rate 1 → 0 flips are due to real memory SEU. Chip S12_A8-1 S12_A8-2 S12_C8-2 S12-C8-1 S14_A8-1 S14_A8-2 S14_C8-1 In figure 22 (b), the Shift Register is set to 0, and 1 → 0 flips dominate. Flips in bit-8 (HitBus) are twice as frequent, probably due to specific layout of that memory bit. Figure 23 shows the distribution of SEU/SET bit flip multiplicity in pixel memory of FE-I4B for the fill 7334 (integrated luminosity 0.35 fb −1 , peak luminosity 1.1 × 10 34 cm −2 s −1 ). In figure 23 (b), the Shift Register state was set to the opposite of the memory content, and bit flips are mainly due to SET on the LOAD line. Each pixel has 13 local memory bits, so multiple bit flips per pixel could be a sign of glitches on the common LOAD line. For example, the glitch generated on the LOAD line before the fan-out to 13 LOAD lines of DICE latches (see figure 1) may result in the synchronous bit flip of many bits (up to 13). The distributions are peaked at one bit flip per pixel, which indicates that the glitch on the LOAD line is mainly specific to each DICE latch bit. However the long tail in the multiplicity distribution may be a sign of small contributions from glitches common to several bits. In figure 23 (a), the Shift Register is set to the same value as the memory, so the bit flips are mainly due to true SEU.

Global register configuration read-back measurements
During LHC fill 7333 the values of the Global Configuration Memory were read back several times and compared with the initial setting. In figure 24, the cumulative fraction of SEU/SET bit flips in the Global Configuration Memory of FE-I4B is shown as a function of integrated luminosity. The high rate of 0 → 1 flips is probably due to SET on the LOAD line with register value 1. The pattern 0xFFFF was loaded last into the register. The values of the register are refreshed several times during the fill. No 1 → 0 real SEU transitions were observed during the fill due to the triply redundant memory design.

SEE cross sections
For the calculation of SEE cross sections, we use the average prediction of the flux of hadrons with energy above 20 MeV 5 in the two outer 3D IBL modules: 91.0 × 10 11 cm −2 fb −1 [12]. This flux was calculated from 13 TeV proton-proton minimum bias events generated using PYTHIA8 [10], applying the A3 6 tune to ATLAS data, and using NNPDF23LO Parton Density Functions (PDF) [19]. The flux of hadrons in the ATLAS detector was simulated using FLUKA version 2011.2x particle transport code [11]. The systematic error on the hadron flux was estimated to be ∼ 30% [12].
In table 1, we summarize the measured rates and SEE cross sections. The rates presented are averages over all available chips and 12 pixel memory bits, excluding the HitBus bit (which shows abnormal rates related to the special layout interconnecting all pixels in the matrix). The systematic errors on the SEE rates were calculated from the RMS spread of the rates in different IBL chips and in different pixel memory bits for SEU and SET dominated samples. SET cross sections are greater than SEU cross sections by a factor of 10. Some differences in 0 → 1 and 1 → 0 transitions are observed, but fall within the present systematic errors. These variations are related to differences in the geometrical positions of the chips, differences in process variations, and differences in the 5The choice of 20 MeV hadron energy cut was justified in [18] by effective reduction of SEU effects below this energy.
6A3 is one of the tunes of PYTHIA8 Monte-Carlo generator to the ATLAS LHC data.
-20 -   using the 24 GeV proton PS beam is recommended, and special attention should be paid to the optimization of chip design not only against SEU, but also against SET effects.

Recovery of bits flipped by SEU
Data that is being assembled into an event fragment from multiple sources becomes de-synchronized when the constituent components actually belong to different events. Such de-synchronization can happen at module level (between FEs), or at readout level (between modules). For this reason, every 5 s, the ATLAS Central Trigger Processor (CTP) sends an Event Counter Reset signal (ECR) inside a 2 ms time window without triggers. The ECR command gets propagated from the various subdetector readout systems to the corresponding FEs in order to regularly re-synchronize the entire readout chain. The IBL DAQ system exploited the 2 ms window to perform regular reconfigurations of the FE-I4B registers; only the global registers were reloaded regularly, while the reconfiguration of single pixel registers was only tested in a few runs. A short description of the software/firmware procedure is described in sections 6.1 and 6.2.

Global register reconfiguration
gle event upsets affecting its global registers as well as the amongst other things loss of occupancy, noisy pixels, and single event upsets as well as the operational issues and o, it is proposed this purpose, ixel front end is process [4].

A. DICE Structure
The DICE latch structure is shown in Figure 1. It is based on the conventional cross coupled inverter latch structure. The 4 nodes X1 to X4 store data as 2 pairs of complementary values. front-end is done via two commercial SNAP12 4 RX modules while configuration data are sent either by a SNAP12 TX module or a TX plugin like the ones used on the current BOC. The Spartan-6 devices will replace all the functionality of the former custom made hardware (BPM encoding, synchronization, variable delays). The S-Link connection is established via Small Form-factor Pluggable 5 (SFP) transceivers. The configuration of the two Spartan-6 devices is handled by a third Spartan-6 (LX75T). As the IBL will return a balanced 8b/10b encoded stream at 160 Mbit/s per link, the new receiver circuitry can implement several advantages over the previous system: • AC coupled receiver logic: due to the balanced encoding, the receiver circuitry can run AC coupled. It therefore delivers a bias free stream, automatically adapting to subtle changes in the data transmission line.
• Transition based code: regular transitions within the received stream allow to automatically synchronise the received data with a local clock, embedding standard synchronisation techniques into the BOCs central programmable device.
• Bytewise encoding: the receiver can be split into an 10b/8b decoder, delivering bytewise output and a formatting section, knowing the frontend data coding scheme. Therefore the individual logic blocks will get simpler in structure and easier to debug.
As for the detector link, the S-Link on the BOC will be implemented directly on the FPGAs and will support twin-channel operation to provide the data stream also to a Fast Track Trigger [9] when needed. In the BOC prototype one of the Spartan-6 connects to a single 1 × 4 SFP cage instrumented with 4 MGTs 6 while the second one connects to a quad opto-transceiver (QSFP 7  -22 - The first implementation of periodic reconfiguration every 5 seconds on the IBL FEs was officially released in ATLAS data taking in August 2017, when the instantaneous luminosity was up to 1.6 × 10 34 cm −2 s −1 . In this version, only the global registers of the IBL FEs are reloaded. A full schematic of the reconfiguration action is shown in figure 25. This action is taken as soon as an ECR signal sent by the CTP reaches the IBL readout electronics. In particular, each IBL ROD is responsible for detecting the ECR signal and propagating it downstream to the corresponding portion of IBL FEs. During the next 1 ms period the ROD inhibits the actions of the CTP signals on the FE-I4B serial lines, including the Bunch Counter Reset signal (BCR), which is broadcast by the CTP every LHC orbit (11 kHz frequency). At the same time, the BOC card, which provides the optical communication interface between the ROD and the modules, detects the ECR and appends to it a sequence of configuration commands, one for each of the 36 FE-I4B global registers. The BOC card has the ability to store the values of each single GR for each FE-I4B in its FIFOs, and it can send a customized stream to all of the modules in parallel. The on-ROD software (running in the PPC), must load the BOC FIFOs at the beginning of each run or whenever there is a change in the module configuration.
The entire procedure of reconfiguration must not last more than 1 ms, since the ATLAS CTP will start triggering again after this time interval. A total of 110 µs is needed to complete the global register reconfiguration.
The main positive effects of the the global register reconfiguration (LV current and hit occupancy stability) have been described in section 1 and are visible in figure 3. Furthermore, this procedure significantly reduces the dead time in ATLAS caused by FE-I4B misconfiguration. Following the implementation of the global register reconfiguration procedure, the occurrence of drops and spikes in the LV current is dramatically reduced. Figure 26 shows that no drops or spikes are observed after relative fill number 22 (LHC fill 6090, 16 August 2017), when the procedure was fully implemented. 7

Single pixel latch reconfiguration
The continuous FE-I4B global memory reconfiguration at ECR has been demonstrated to be very effective for ATLAS data taking. For this reason, the regular reconfiguration of single pixel latches was also investigated. Unfortunately, there is not enough time in the ECR time window to reconfigure all pixel latches at once.
The solution implemented addresses up to three latches of one pixel double column at each ECR. In total the FE-I4B hosts 13 pixel latches and 40 double columns; only 11 out of 13 latches (Enable, HitBus, TDAC and FDAC) are regularly reconfigured because of their direct impact on noise and quiet pixels. Any relevant pixel latch is reconfigured every ∼130 ECRs (meaning a full reconfiguration every ∼11 minutes). The HitBus latch is actually refreshed more often than others, since a uniform value must be loaded in the entire chip; this bit can indeed be loaded in parallel in all the pixels using a different register writing modality. Unlike the global reconfiguration, a different BOC FIFO content is required at each ECR, since a different group of latches and a different double column are addressed every time. This implementation required a more intense use of the on-ROD PPC. The automatic reconfiguration of the pixel latches was carried out only in few test runs in July 2018 for a relevant portion of the detector (all the IBL FEs located at η = ± 2.8, corresponding to IBL DAQ modules in position A7 and C7). Unfortunately, due to some instabilities in the on-ROD PPC software, this implementation was only used for test runs and was not fully deployed for the remaining LHC Run 2 period.  Figure 27. The noise occupancy in IBL 3D modules as a function of integrated luminosity with and without the mechanism of pixel register reconfiguration at ECR. 7Current drops (or spikes) of 1.5% in four chip modules are counted if they last longer than 100 seconds and occur during LHC fills longer than four hours. The first and the last half hour in each fill are excluded to avoid edge instabilities. To suppress current drops (or spikes) due to ROD configurations and luminosity fluctuations, we also require the drop occur in only one IBL module during in a ± 20 min interval. Figure 27 shows the noise occupancy (the number of hits per pixel per event) in IBL 3D modules as a function of integrated luminosity with and without the mechanism of pixel register reconfiguration at ECR timing. The mechanism is adopted for modules at A7 and C7 and is not deployed for those at A8 and C8. Figure 27 is made from the calibration PixelNoise stream, where random data are taken with empty, isolated bunches. The data points include statistical error bars. Without the mechanism of pixel register reconfiguration, SEUs cause an increase in the noise occupancy during the run. When the reconfiguration mechanism is adopted, the noise decreases as a function of integrated luminosity. This is because the reconfiguration is performed every 11 minutes for the same pixel registers, but the time required to accumulate the same amount of integrated luminosity becomes longer during the run due to the decrease of instantaneous luminosity.  ) shows the fraction of quiet (noisy) pixels as a function of integrated luminosity in the eight IBL η rings in the LHC fill 7018 of 2018. The data are extracted from randomly triggered events, as discussed in section 4.2. In the modules without regular reconfiguration, more pixels become noisy or quiet during data taking due to SEU. In the modules reconfigured every 11 minutes, the fraction of noisy or quiet pixels is more stable. Figure 29 shows the fraction of broken primary clusters as a function of integrated luminosity in fill 7018 from 2018, in the eight rings of 3D modules. The 14 modules in the same ring have been studied together to achieve higher statistics. The four rings of 3D modules without regular reconfiguration show a similar increase in the rate of broken clusters. The four rings of 3D modules with regular reconfiguration show a similar decrease in the rate of broken clusters. This is because the full reconfiguration occurs every 11 minutes, and more time is needed to accumulate the same amount of luminosity at later times when the instantaneous luminosity is lower. The modules on IBL stave 6 were reconfigured after ∼320 pb −1 of data were collected. This corresponds to the changes of slope in the modules without reconfiguration. Figure 30 (a) shows the average hit ToT from single-pixel clusters as a function of integrated luminosity in the LHC fill 7018 from 2018, in the eight 3D IBL η rings. In the modules without regular reconfiguration, the average ToT increases because more pixels become noisy due to SEU. In the modules reconfigured every 11 minutes, the average ToT is more stable. To confirm that the -25 - increase of ToT is due to noisy pixels, the average hit ToT is studied with noisy pixels excluded (noisy pixels are identified in figure 28 (b)). Figure 30 (b) shows that the average ToT in the modules without regular reconfiguration is more stable after excluding noisy pixels, which proves that the increase of ToT during the fill is almost fully correlated with noisy pixels caused by SEU in TDAC MSB. The detuning of pixels due to SEU in FDAC causes the ToT to get smaller or larger. Due to the dominance of 0→1 transitions we expect an increase in FDAC values more often than a decrease, but the effect is small and not visible on figure 28 (b). In the four rings without reconfiguration we observe the increase of the standard deviation, probably due to the flips of FDAC bits.

Conclusion
The ATLAS Pixel detector, including the IBL, was operated efficiently at high luminosity, and SEUs were observed at rates close to the expectation based on previous tests in 24 GeV proton beam. SEUs in the global and local pixel configuration memories of FE-I4B chip were observed during LHC fills and had several consequences: module de-synchronization, current jumps, dead modules, quiet pixels, noisy pixels, broken clusters, etc. By read-back measurements and simulations of the electronics, it was demonstrated that SETs on the LOAD line of the DICE latch dominate the memory flips. These SETs explain the apparent asymmetries of 0 → 1 and 1 → 0 transitions.
Starting in 2016, some of the effects of SEU/SET were mitigated by occasional manual or automatic reconfiguration of problematic modules. The impact of SEUs on the global configuration was mitigated in 2017 by refreshing the global memory every 5 seconds; this was syncronized with ECR and therefore resulted in no additional dead time to ATLAS. It was demonstrated in a few test runs in 2018 that gradually refreshing local pixel memory during ECR reduces the number of noisy and quiet pixels and broken clusters. We plan to introduce this local memory refreshing in all IBL modules in 2021 for LHC Run 3 and therefore improve the performance with potential future increases of the LHC luminosity.
The operational SEU/SET experience with FE-I4B was also used to inform the design of the next generation of LHC pixel front end chip FE-RD53 [20], [21]. High frequency refreshing of all global and local memories (sometimes referred to as trickle configuration) is embedded in the design of this chip. The RD53 command protocol permits sending configuration and sub-circuit reset instructions concurrently with data acquisition triggers, without incurring dead time. Figure 32. Schematic for a full stave of the module and FE-I4B layout, as well as the DCS/power system modularity. The FE-I4B chips are labelled as A(C)X-Y where X runs from 1 to 8 towards the high |η| direction, and Y is 1 or 2. A(C)7 − 1/2 and A(C)8 − 1/2 are single-chip 3D modules. The DCS modularity is shown as M1A(C) through M4A(C). The NTCs measure the temperature of each DCS module group.

A Description of IBL detector notations
In this appendix we will give a short description of the IBL detector in terms of geometric layout, power system (LV and HV) and data acquisition (DAQ) granularities. A full description can be found elsewhere [6].
To indicate a specific IBL DAQ module, the following name scheme is adopted: LI_SXX_A/C_MYY_A/CZZ where • XX refers to the stave ID number, • A/C refers to the position of the module in the longitudinal axis with A → z > 0 and C → z < 0, • YY refers to the DCS module group and goes from 1 to 4 depending on the position of the module along the stave, • ZZ refers to the DAQ module that goes from 1 to 8 depending on the position of the module along the stave.
A graphical representation of the naming scheme is shown in Fig. 32. One example of naming scheme for a DAQ module is LI_S02_C_M4_C8 In case a single FE needs to be addressed, an extra 1/2 is append to the end of the module name.
One example of the naming scheme for a single FE is LI_S02_C_M4_C8-2.
In this document, DCS module groups (powering 4 FEs with the same LV and HV), DAQ modules (controlling and reading out 2 FEs), Sensors (connected to 2 or 1 FEs depending on the planar or 3D technology) or single FEs will be used depending on the granularity affected.