Impact of magnetic field on the stability of the CMS GE1/1 GEM detector operation

The Gas Electron Multiplier (GEM) detectors of the GE1/1 station of the CMS experiment have been operated in the CMS magnetic field for the first time on the 7th of October 2021. During the magnetic field ramps, several discharge phenomena were observed, leading to instability in the GEM High Voltage (HV) power system. In order to reproduce the behavior, it was decided to conduct a dedicated test at the CERN North Area with the Goliath magnet, using four GE1/1 spare chambers. The test consisted in studying the characteristics of discharge events that occurred in different detector configurations and external conditions. Multiple magnetic field ramps were performed in sequence: patterns in the evolution of the discharge rates were observed with these data. The goal of this test is the understanding of the experimental conditions inducing discharges and short circuits in a GEM foil. The results of this test lead to the development of procedure for the optimal operation and performance of GEM detectors in the CMS experiment during the magnet ramps. Another important result is the estimation of the probability of short circuit generation, at 68 % confidence level, p short HV OFF = 0.42-0.35 +0.94% with detector HV OFF and p short HV OFF < 0.49% with the HV ON. These numbers are specific for the detectors used during this test, but they provide a first quantitative indication on the phenomenon, and a point of comparison for future studies adopting the same procedure.


Introduction
During the Long Shutdown 2 (LS2), the muon system of the Compact Muon Solenoid (CMS) experiment was upgraded with the installation of a new triple-GEM detector (where GEM is Gas Electron Multiplier) muon station, GE1/1 [1].
The GE1/1 system consists of 144 chambers, paired to form 72 super-chambers (36 per endcap).Two kinds of GE1/1 super-chambers alternate in the azimuthal plane between long (GE1/1-L) and short 36 (GE1/1-S) versions as in figure 1, which are dictated by the mechanical envelope of the existing endcap.The trapezoidal chambers with an opening angle of 10.15 • are installed in a staggered arrangement with an overlap of 2.6 mrad.The dimensions of the chamber are shown in table 1.
Trapezoid parameter (active volume) GE1/1-S GE1/1-L Longitudinal length (cm) 106.1 120.9 Short basis (cm) 23.1 23.1 Long basis (cm) 42.0 44.6 Thickness (cm) 0.7 0.7 The purpose of the GE1/1 detector is to measure the muon bending angle in the pseudorapidity region 1.55 < || < 2.18, measured by a GE1/1 chamber and the adjacent Cathode Strip Chamber (CSC).This reduces the Level-1 trigger rate, keeping it under control, as made necessary by the new challenging conditions expected in Run-3 and in the High Luminosity runs of the Large Hadron Collider (LHC) [2].The design requirements include: maximum geometric acceptance, rate capability of 10 kHz/cm 2 , single-chamber efficiency of at least 97%, angular resolution of 300 µrad, timing resolution of 10 ns, and gain uniformity of 37% [16].
-1 -M chambers form a superchamber.Right: Long and short chame the instrumentation within given mechanical constraints in the set, Versatile link and GLIB/MP7 μTCA systems can provide the munication system required.treated as an individual unit from an electronics system point of gmented in both φ and η; the baseline for LS2 is segmentation of ng a maximum of 24 individual detector segments.Each of these d into 128 strips and read out by one 128-channel front-end chip.ntly has up to 24 front-end chips and channels organised in three ed such that one optical fibre can read out the tracking data from gger data are carried out by a dedicated additional fibre.A single al fibres to take the tracking and trigger data to and from the CMS from the VFAT chips are sent to the GEB which delivers power and from the VFAT hybrid as well as providing the connection GEB, data are transmitted to one FPGA board, called the GEM the wide end of the GEM module.One of the main components FPGA, which has been shown to be radiation-hard to levels at higher than the expected radiation dosage.[3].Highlighted in blue (light) are the long detectors and in red (dark) are the short ones.Reproduced from [3].CC BY 4.0.
A triple-GEM detector consists of three cascaded GEM foils, henceforth referred to as GEM1, GEM2, and GEM3 from the cathode to anode, and four gas gaps.A scheme illustrating the electrodes stack in triple-GEM detectors is reported in figure 2, and the dimensions of the thickness of the four gas gaps are summarized in table 2.
A GEM foil of GE1/1 consists in a 50 µm thin polyimide foil with 5 µm copper-coated faces perforated with high density microscopic holes [4].The holes, obtained by chemical etching, are truncated double cones with outer and inner diameters of 70 µm and 50 µm respectively; they are spaced with a pitch of 140 µm in a hexagonal pattern.The triple-GEM cascade can achieve a high total amplification, up to 10 5 , with a modest high voltage (∼ 400 V) across each of the three individual foils thus avoiding electrical breakdown.Each of the three GEM foils are identical and trapezoidal shaped as the detector.The GEM foil surfaces oriented towards the readout board (GEM foil bottom face) are a single contiguous conductor while the surfaces oriented towards the drift board (GEM foil top face) are segmented in 40 High Voltage (HV) sectors for the short detectors (47 HV sectors for the long detectors), parallel to the base of the trapezoid providing longitudinal segmentation.The width of each HV sector is adapted moving from the short base to the wide end to obtain an equal area for each sector of ∼ 100 cm 2 .This segmentation has been introduced -2 -to reduce the charge flowing through one foil during a discharge, limiting the involved energy , because of the smaller capacitance  of the single HV sector  = 1 2 Δ 2 .If a discharge were to occur in one sector, the damage would only involve that sector preserving the rest of the detector.In addition, this design allows to better keep the desired voltage applied on all the sectors not involved by the discharge event.The top face of the foil is segmented, while the bottom is not: this design reduces the impact of the discharge events on the correct supply of HV to the different detector regions, but contributes to a small drop of the voltage globally on the foil.Figure 3 represents the scheme adopted to apply HV to the foil sectors.• HV, to enable transport and multiplication of electrons produced in the gas medium by the traversing muons.For GE1/1 detectors, HV is provided by a multichannel power supply board (CAEN A1515BTG [10]).This allows the seven electrode voltages to be set -3 - independently.In particular it applies a potential difference between two detector planes and then monitors the current flowing in the power supply needed to maintain the potential difference applied (figure 4).• Low Voltage to power the readout electronics is provided with CAEN A3016-HP boards [11];

GEM1 GEM2
• cooling water, for electronics refrigeration, supplied with a temperature of 16 • C and a pressure of 9 bar.
To correctly operate a GE1/1 detector, electric fields are generated: • in the gas region filling the GEM holes, by applying a voltage difference between the top and bottom faces of the GEM foil; • in the gas gaps, by applying a voltage difference between the faces of the upper and lower planes in the stack.
This powering scheme is illustrated in figure 4, reporting the naming convention used to identify the channels of the A1515 power supply feeding a specific detector component.
For the sake of clarity, in the following we briefly explain how the voltages to be applied to the individual channels are defined.Assuming a voltage divider such as the one shown in figure 5 is used for the power supply, it is possible to derive the voltage configuration using a single parameter, i.e. the current flowing through the divider,  eq .Given that value, the voltage to be applied to each channel in the stack can be determined.The choice originated from the R&D phase for CMS GEM, during which the detectors were actually fed through a resistive divider such as the one in figure 5.During this phase, the divider served in particular for gain measurements: this convention, labeling a voltage configuration in terms of  eq , is so useful to have a unique number for performance comparison of detectors and has so been maintained in time.Example configurations are reported in table 3: the most common  eq voltages configurations adopted for GE1/1 detectors operating in CMS are 690 and 700 µA.In addition, table 4 reports the electric fields generated in the foils and in the gas gaps, corresponding to each  eq configuration.
-4 - The A1515 board applies a potential difference between two detectors planes and then monitors the current flowing in the power supply to maintain the potential difference applied.In addition, the naming conventions of electrodes when monitored by a 7-channel pico-ammeter is also reported.Differently from A1515 board, this device monitors the current flowing on each plane of the stack.To complete the introduction to GE1/1, figure 6 shows its position in the CMS experiment and the intensity of the magnetic field generated by the CMS magnet (| ì | = 3 T at the location of the GE1/1 detectors [3]).The magnetic field lines in the position where the GE1/1 station is installed -5 -

Anomalous trips and shorts observed in CMS during magnet ramp
In the final stages of the LS2, the CMS experiment powered its superconducting magnet.The operation started on the 7 th of October 2021, with the first ramp up of the magnet since the beginning of LS2.The magnetic field ramp lasted for 11 minutes, reaching a maximum magnetic field value of -6 - 0.2 Tesla (T) (figure 7).In this figure, all the magnet ramps performed in one month time scale are illustrated; it is worth to note that the initial ramp consists of a few intermediate steps between 0−0.4 T.

JINST 18 P11029
During the first magnetic field ramp, the HV system of GE1/1 detectors was ON and supplying a set of voltages equivalent to a divider current  eq = 690 µA (table 3), but keeping OFF the voltage applied on the induction gap, meaning that no potential difference is applied on this single gap.This one is indeed the gap closest to the readout plane and was turned off to protect the electronics in case any anomalous event occurred during the first CMS magnet ramp.
During that ramp, 60 GE1/1 super-chambers out of 72, experienced instabilities on the HV system, consisting of the occurrence of positive spikes in the monitored current ( mon ).These spikes triggered a protection switch-off of the involved channels due to the overcoming of the safety limit set on the monitored current ( 0 threshold), for a time longer than the time duration parameter ( trip ).Such events are called HV trips.During the ramp, the two protection parameters were  0 = 2 µA and  trip = 1 s.
Earlier during the GE1/1 commissioning, an automatic trip recovery procedure was implemented in the GEM Detector Control System to handle the trips.This procedure took action also in this case, re-powering the HV of a detector after a trip event.At power up we discovered two new short circuits, in a GEM foil of two distinct chambers.During the HV training procedure of the foils at 430 V, one of this two healed.For this reason, in the following we will count only one short circuit for this ramp up.
A short circuit in a GE1/1 foil consists in a connection created between the top and bottom faces of the GEM foil.This connection can be generated, for example, by copper initially deposited on a face of the foil, which melts due to a damaging event (for example a discharge) and deposits inside the hole.Every time this happens, the voltage difference between the top and bottom faces -7 -of the affected foil drops to 0 V, deactivating the electric field inside the GEM foil holes.For this reason, the GE1/1 GEM foils are divided in sectors, and the HV power is provided according to the scheme in figure 3.This structure allows the GE1/1 detectors to maintain the field active in the sectors not affected by the short circuit.
If a short circuit is generated in a GEM foil, the equivalent resistance seen by the power supply is no longer that of the GEM foil (O (Ω)) , but only the equivalent resistance of the resistors placed on the path generated by the short circuit, as can be understood from figure 3.Because of this, a short circuit leads to an increase in the current flowing in the channel of the power supply powering the involved amplification region.In addition, the voltage difference falling across the protection resistors leads to the application of a reduced voltage (∼ 6% lower) on the faces of the foil, respect to what was provided by the power supply, because of the higher current flowing through the HV filter.This drop can be calculated from the voltage falling across the resistors of the equivalent circuit created by the short circuit, between the top and bottom faces of a GEM foil (figure 3).Otherwise, when no short circuit is present in the foil, the current flowing in the power supply channel is lower than 0.2 µA which translates in a negligible voltage drop on the protection resistor with respect to the nominal values.
In response to the observation of the large number of trips (60) during the first ramp of the CMS magnet, the GE1/1 HV system was turned off.
In the following days, new magnetic field ramps were performed and GEMs were OFF.During the periods with a stable magnetic field, an HV training procedure was performed for GE1/1 detectors.This procedure consists in applying a fixed voltage value for an extended time period, varying between 13 and 42 hours, to one foil at a time for all the detectors in parallel.The purpose was to monitor the single foil trip rate and to burn any residual dust particle that might have accumulated on the foil at any step after the assembly, which could have been responsible of triggering a trip event.
During the days following the first magnet ramp, other short circuits were created.In particular: • 2 appeared on October 11 th , after activating one foil following 3 small magnetic field ramps ( max = 0.22 T) performed with detector HV off; • 1 on October 20 th , during the training of all the 3 foils at 430 V, with the nominal magnetic field stable at  = 3.8 T; • 1 on October 29 th , formed during the operation of the detector with HV active on all 7 electrodes, with the nominal magnetic field stable at  = 3.8 T; • 7 from November 2 nd to 5 th .In this case, a complete ramp down of the CMS magnet was performed on November 1 st , from 3.8 T to 0 T.During the ramp, the HV on all chambers was kept OFF.Then 6 shorts were discovered on November 2 nd as soon as the 7 HV electrodes were powered on, and an additional 1 on November 5 th .The evolution of the CMS magnetic field in the period of interest can be followed in figure 7.
These short circuits added up to the other already present in the chambers before the magnet ramps.The pie charts in figure 8 show the distribution of HV short circuits as a function of the operating conditions (figure 8a) and the GEM foils affected (figure 8b).These plots provide a summary of all the short circuits present in GE1/1 detectors up to 5 th November 2021, also those generated before the operations of the CMS magnet.In particular, regarding figure 8a, the operating conditions of the detectors present at the moment of a short circuit creation are: -8 - • Disk movement: movement of the CMS disk where GE1/1 is installed; the detector HV is OFF during that operation (1 short circuit generated); in CMS a disk is a vertical structure hosting several detector stations positioned at the same z coordinate in the CMS frame of reference; • HV training, B= 0 T: ongoing HV training procedure for GE1/1 detectors, with CMS magnet OFF (4 short circuits generated in this context); • HV training, B= 3.8 T: ongoing HV training procedure for GE1/1 detectors, with CMS magnet ON and generating a stable magnetic field (maximum field  = 3.8 T, and  = 3.0 T in GE1/1) (2 short circuits generated in this context); • B first ramp up: magnet ramp up, with HV ON: first time GE1/1 detectors experienced magnetic field variation (1 short circuit generated).A second short was also generated during this magnet ramp, but the HV training of the following days was able to heal this short circuit; • Change mode SB → RD, B= 3.8 T: change of the detectors' HV working point from STANDBY mode  eq = 580 µA to the READY FOR PHYSICS mode  eq = 700 µA.The CMS maximum magnetic field was stable at  = 3.8 T (1 short circuit generated).The exact meaning of these two operating states is explained in section 3. To understand the occurrence of HV instabilities and the creation of short circuits, the CMS GEM group decided to perform a test in CERN North Area with a 1.6 T magnet on spare GE1/1 chambers.The aim of the test was to understand which are the relevant parameters in the discharges and shorts development during magnetic field ramps, and to define a safe procedure to operate GE1/1 detectors every time a magnet ramp is performed.

Discharge test in high magnetic field
A magnet able to provide a high field was used to reproduce and study the observations made during GE1/1 operation in CMS detector.The Goliath magnet at CERN North Experimental Hall-1 is able to reach uniform fields up to 1.6 T in the volume between the two 1.5 m diameter coils [9], as reported in figure 9a.

Experimental setup
The test with the Goliath magnet made use of four triple-GEM detectors, three from mass production of GE1/1 station [16], and one from the GE1/1 demonstrator that was installed on CMS between 2017 and 2018 [17].The experimental setup consisted in: • two long and two short GE1/1 detectors operating in Ar/CO 2 (70/30); • a light and versatile stand in aluminum bosh-profiles able to allocate a maximum of six detectors (figure 9b); • two CAEN A1515 HV boards mounted in a CAEN SY4527 mainframe [12].Each board is able to power 2 detectors independently and to monitor voltage (resolution up to 20 mV) and current (resolution up to 400 pA) from each electrode of a single detector with a sampling rate of the order of 1 Hz; • one Metrolab THM1176-HF magnetic sensor for 3D magnetic field monitoring with a resolution of 0. -11 -

Test objectives and procedure followed in the test execution
The main three goals of the test were: 1. reproducing the appearance of the shorts during magnet ramps with detectors OFF; 2. finding a safe way to change the voltage applied to the channels with a stable magnetic field; 3. investigating factors playing a role in the discharges recorded during magnet ramps when detectors are ON.The first item was investigated by performing magnetic field ramps with the detector OFF and checking for shorts afterwards by applying 50 V on the three GEM foils.
The second item is related to the operational necessity of changing channel voltages during different phases of the LHC cycle: in fact, during a single fill, the detectors must switch from a safe -12 - low-gain or even zero-gain condition (STANDBY mode) during the more delicate phases of the beam injection, to a fully efficient condition (READY_FOR_PHYSICS mode) during collisions.The STANDBY to READY_FOR_PHYSICS transition, in our case, consists in changing the detectors working points from 580 to 700 µA, with voltages set accordingly to table 3. Another test was performed by setting the magnetic field at its maximum stable value of 1.6 T and performing the transitions with different HV ramping-up configurations.The HV ramping order was changed among two types: a first one, called GEM mode, where the HV on all the seven channels is raised at the same time, and a more conservative one, called CMS experimental site mode, where the channels are turned ON one by one, first all the foils and then all the gas gaps.The ramp speed was kept to 2 V/s on all the gaps except on drift and transfer2, where values of 5, 4, 3 V/s and 4, 3 V/s were respectively investigated.
The third item was the primary goal of this test.A multitude of magnet ramps was performed with the detector ON to investigate the factors that play a significant role in the occurrence of a discharge.The variables changed during the test were: • the detector working points: 700, 710, and 720 µA were applied to understand if a higher amplification field can increase the discharge rate.• gas flow: the considered values were 5, 7, 10, and 12 L/h to understand if the over-pressure on the detector can influence the discharge rate and if a higher flux can move more dust inside the chamber.
-13 - • intensity of the magnetic field: nominal fields were changed between the range [0.2, 1.6] T.
• direction of the magnetic field: the first 30 ramps were made with a fixed direction of the field, while the rest of the test also exploited changes in the field direction.This allows to investigate possible asymmetries on the two CMS disks.• mechanical stress: the detectors mounted on the aluminum profile structure were shaken to simulate a CMS disk movement, which can induce dust movement inside the gas volume.
One gentle shake was performed when the structure was just crawled on the magnet surface, while a second rough shake was performed when the structure fell from around 10 cm above the magnet floor.• detector orientation: during the last part of the test, the detector structure was rotated by 90 • , having the detector vertically in the same direction of the magnetic field.During the test, monitoring plots were produced both for data collected from A1515 and the pico-ammeter to observe the evolution of discharges in the different electrodes.The two instruments provide complementary information to diagnose and characterize the discharge event.The A1515 is fundamental to have an actual reference of the behavior of the power system for the GE1/1 detectors installed in CMS; on the other hand, the pico-ammeter offers a much higher resolution in time and current of the same event, potentially allowing to understand better its evolution.
An example of discharge as detected by the A1515 board is illustrated in figure 10.Four plots in total are presented: on the right of the monitored currents ( mon ) and on the left of the monitored voltage.The 7 electrodes powering the detector have been separated into two distinct plots for clarity of display: • Foils: GEM1, GEM2 and GEM3 (figures 10a-10b); • Gas gaps: drift, transfer1, transfer2 and induction (figures 10c-10d).The nomenclature used in the plots, for both A1515 and pico-ammeter data, has been presented in figure 4, which indeed shows as: • the currents read by the A1515 are those observed in the power supply corresponding to the actual voltage difference applied between two given planes of the detector, as explained in detail in ref. [10]; • the currents read by the pico-ammeter are the currents flowing on a given plane of the detector.The A1515 plots reported in figure 10 show the data collected from a single chamber during a single ramp down of the magnetic field generated by the Goliath magnet when the detector was operating at a voltage configuration equivalent to a divider current of  eq = 700 µA.Figures 10a  and 10c show the voltages observed in the chamber, while figures 10b and 10d the currents observed on the foils and the gas gaps respectively.
In particular, in the voltage plots a complete turn off of all the 7 HV electrodes can be observed.This turn off event, also called trip, was caused by a spike in the monitored current, visible in the current plots.The spike in current was indeed high enough to trigger the protection turn off of the HV power, since it overcame the current safety limit  0 , set at  0 = 2 µA, for a time longer than  trip = 1 s.These spikes in current are due to discharges produced inside of the detector, likely starting from the edge of a GEM hole.
The following negative currents observed in the plot are due to the voltage ramp down of all the electrodes.Similarly, the following positive currents are due to the recovery ramp up of the voltages of the 7 electrodes.
-14 -The significant aspect of the observed discharge event is that it occurred during the variation of the magnetic field and not during its stabilization at a fixed value.This is a common feature of the discharge events observed during the test.Figure 11 shows an example of a spike observed through the data recorded by the pico-ammeter (this event is not the same displayed in figure 10).
The plot shows the spike corresponding to the discharge, occurring simultaneously on each of the GEM foils, and the subsequent charging up of the foils.From the current plots, an exponential shape can be observed compatible with the charging up of an RC circuit.A GEM foil could be seen as a capacitor connected in series with the protection resistance and the HV filter.Due to the components in the filter the whole system can be approximated as an RC circuit in which the mainly components are the protection resistance and the GEM capacitance.If this approximation is correct, the slope of the exponential shape has to be compatible with the product of the resistance and capacitance values of the GEM foil.To test this hypothesis, first a static measurement of the capacitance is performed with a multi-meter keeping the HV OFF on the detector.Then the current curves, shown in figure 11 are fitted with an exponential, and the resulting capacitance is compatible with the static measurement.Through the exponential fit and considering the protection resistance of 10 MΩ, a capacitance value of 15.345±0.005nF has been obtained, where the statistical uncertainty is reported.This gives confidence that such discharge events are not damaging the detector, given the foil manages to fully recharge.
This measurement is made possible due to the high sampling frequency of the pico-ammeter, while the sampling time of A1515 is much coarser.The difference between the current values in figures 10 and 11 are explained in section 1.
Moreover, the purpose of the two measurements, with the pico-ammeter and the A1515 is different.As mentioned before, the pico-ammeter data allows to follow the charging up of the foils after the voltage drops due to discharge and to identify discharge structure.The A1515 data instead can be used to establish the protocol to be applied at CMS to mitigate discharge events.As mentioned in section 1, the GE1/1 detectors at CMS are read through A1515 boards, and therefore the required response in term of  0 and trip times should be set on the same type of board.The current values measured by the two instruments can be compared taking the differences of the pico-ammeter currents values as shown in figure 4, but for the scope of this paper, this re-evaluation is not necessary because the reference value for the HV power turn off depends only from the A1515 monitoring.
The data collected by the pico-ammeter therefore allow a clear identification and characterization of discharge events.

Results
All the proposed tests were successfully carried on and the results are reported in the following.Short circuit generation.Among the 60 ramps done with the detector HV OFF, we observed the generation of one single short circuit in one foil of one detector.From these results, assuming the same generation probability of short circuit per magnet ramp, a short generation probability per ramp per detector with HV OFF is calculated to be equal to 0.42 +0.94 −0.35 %, at 68% confidence level (CL), estimated with the Clopper-Pearson approach [19].The total number of trials is the total number of ramps performed with HV OFF, multiplied by the number (4) of detectors involved, so  = 240.
-15 - The short created in such a way was cured by applying 500 V for less than one second on the affected foil with an insulation tester with the detector flushing in the Ar/CO 2 standard mixture [8,16].
No short circuits were detected during ramps with detector HV ON, resulting in a generation probability per ramp less than 0.49% with the same CL.The total number of ramps performed on the 4 detectors (the number of trials) was  = 372.
Transition mode.Another test was aimed at investigating the safeness of the STANDBY to READY_FOR_PHYSICS mode transitions.During the tests, no discharge was observed in any of the proposed configurations.The discharge probability per each transition has been calculated using a binomial distribution in the assumption of equal probability for each transition, where the number of trials is the number of HV transitions performed.Grouping the transitions in GEM mode ( = 32) and CMS experimental site mode ( = 30), the upper limits for discharge probability per transition have been calculated to be respectively less than 5.57% and than 5.93% at 68% CL.
Measurement of discharge rate.Finally, a total of 89 magnet ramps were made with the detector ON, operating in different conditions of HV, gas flow rate, and magnetic field.All the discharges recorded during the test were correctly tagged by both the A1515 board and the floating picoammeter.Figure 12 shows the moving average on 5 ramps of the discharges per ramp per detector as a function of the run number (one ramp was done for every run).No distinction among the magnetic field intensities is made since no clear trend was identified.The red dashed vertical lines indicate when the field ramps with the detector OFF happened, highlighting the count of consecutive multiple ramps in the same conditions with red markings.The black vertical lines, on the other hand, refer to the time when the two shakes were performed.As mentioned before, during the first 30 ramps, the fields were kept positive, i.e. pointing upwards.A red line at run 30 indicates when the negative field direction ramps start to play a role.
Qualitatively it can be noticed that: • the discharge rate decreases from the beginning of the test to run 17; • no noticeable effect was detected in the first 14 magnet ramps performed with HV OFF; • after a gentle mechanical shake and 12 magnet ramps with HV OFF (one of which caused a short circuit in one detector), a slight rise of the discharge rate is detected, but it returned compatible to zero after 7 more ramps.• no more discharges were then observed until the first ramps with opposite magnetic fields were performed; • when ramping with inverted magnetic field, a rise of the discharge probability was detected.
This increase in the discharge rate returned compatible with zero after 7 ramps; • the run 38 was dedicated to ramps with detector OFF (12 in total), with  field in both polarities; • after these ramps, the discharge probability increases up to around the value of the beginning of the test, and it goes back to zero after around 13 ramps, where it stays for a relatively long time (around 20 ramps); • an increase was then detected when the detector was powered at the highest tested working point 720 µA; Evolution of the discharge rate per magnet ramp per detector.The Run number is an incremental number used to identify a data taking time interval, when a single kind of ramp was performed: a single magnet ramp with HV ON, or multiple magnet ramps with HV OFF (identified with red vertical dashed lines accompanied by a red number indicating the count of multiple ramps).The main detector parameters varied during the test were the gas flow and the HV working point, showed in the plot bottom bands.Other situations indicated on the plot are the first inversion of the magnetic field sign (red vertical solid line), the mechanical stresses (black vertical solid line) and the change in the detector orientation (figure 9c) respect to the  field direction (yellow box in the right part inside the plot).During all the previous test period the detectors' planes where orthogonal to the  field direction (figure 9a).
• the rising trend also continued after the rough mechanical shake and started to decrease after a peak; • finally, the detectors were positioned vertically, laying on the long side of the chamber frame, with gravity force parallel to the chamber plane.This arrangement reproduces the direction of gravity force experienced by the chambers in CMS, but not the orientation of the magnetic field.Here a new rise in the discharge probability was observed, followed by a final drop.
-17 -As mentioned before, neither evident influence of the magnetic field intensity, nor influence of the gas flow (from the above discussion) has been identified.The working point has no evident effect on the discharge rate, but a slight increase in the probability is detected at the highest working point.However, because of the high gain in such a configuration (order of 1 × 10 5 ), the increase in the probability due to magnet ramp performed with a given gain, cannot be disentangled from the intrinsic increase of the spark probability of such a gain configuration, i.e. the discharge probability of the GEM detector itself without presence of external factors.On the other hand, it is clear that, at a fixed condition, the detector's discharge probability tends to lower after multiple ramps.Only certain operations trigger a re-ignition of the discharges, in particular: • the mechanical stress such as shaking of the stack, similar to what can happen to the detectors when the CMS disks move; • the inversion of the magnetic field; • the change in detector orientation.Such experimental observations lead the authors to the hypothesis that discharges and shorts during magnetic field ramps are caused by small particles (order of tens of µm) trapped inside the gas volume.Indeed, even if the detectors have been assembled in an ISO-6 clean room [7,14], some dust produced or deposited during the assembly procedure may have been trapped inside the detector or even created after closure of the detector (screwing).Such particles may remain attached to the detectors' surfaces by electrostatic force, and they can be detached by a small force applied by some mechanical stress or the variation of a magnetic field (also in the case of non-ferromagnetic materials).Under this hypothesis, we can explain the behavior of CMS GE1/1 detectors in magnetic fields.Indeed, a dust particle removed from a detector plane may start to float inside the detector and enter a GEM hole of one of the foils.Two different scenarios may occur: 1. if the foil is powered, the dust particle is burned with high current.Indeed, the particle immediately produces a conductive path between the two sides of the foil.The high current flowing through it rapidly evaporates the particle itself.No significant damage is expected to occur [15]; 2. if the foil is OFF, the dust can deposit in the GEM hole, if one slowly ramps the detector to turn it ON, the dust can carbonize/melt, depending on its chemical composition, and potentially create a permanent conductive path, i.e. a short circuit in the foil.Such behavior was observed during the GE1/1 production [16].The final answer on the chemical nature of the dust can be answered only when the detector will be un-installed from CMS and opened.Magnetic field ramps may detach dust particles from the detector surfaces: the removed particles are finally burned by multiple ramps.Changing the magnetic field direction can tear off those particles that were not moved by the force induced by the variation of the magnetic field, respect to their initial configuration.This could explain not only the increase of discharge probability when changing the relative orientation between the magnetic field and detectors, but also because the probability decreases with the number of ramps.Mechanical stress may rearrange and detach the particles, which then become prone to removal by the effect of the magnetic field.
The previous conclusions support the hypothesis that dust trapped inside GE1/1 detectors might be the basis of the observed behavior, both in the CMS detector and in the Goliath test.From this experience, two major considerations about the safe operation of GEM detectors during ramps of magnetic field can be thrown: -18 - Fraction of discharge events overcoming a given current threshold [18].Reprinted from [18], Copyright (2023), with permission from Elsevier.

JINST 18 P11029
• keep the GEM foils at the nominal voltage while the drift, transfers and induction gaps are OFF.This should allow the burning of the dust moving in GEM holes reducing the propagation of discharges towards the readout; • increase the current limit for trips on the GEM foils to several µA to avoid having a trip during the magnet ramp.Indeed, during the ramp down and the eventual recovery of the foil, more dust can deposit.In CMS, the current trip limit during ramp was set at 20 µA.This choice was guided by the distribution in amplitude of discharges observed during the test of chambers at Goliath magnet: a plot showing the fraction of discharges overcoming a given threshold is reported in figure 13; • in addition the time allowed for a discharge to overcome a given  0 current limit, also called trip time parameter ( trip ), was increased from 1 to 3 seconds.This was done to prevent HV trip in case of multiple discharges close in time, that can appear in the A1515 as one single long-duration discharge.This kind of event is rare but can occur, and it was only possible to distinguish from regular discharges thanks to the time resolution of the pico-ammeter.
Such operative procedures were adopted by the CMS GEM Collaboration to operate the GE1/1 station in the CMS experiment.After these actions were implemented during operation in 2022, only 2 more short circuits were generated among the 144 GE1/1 detectors installed in CMS during the 8 magnetic field ramps performed in 2022.In addition, only 3 HV trips were observed in the same number of ramps, uncorrelated to the generation of short circuits.

Conclusions
The test of GE1/1 spare Gas Electron Multiplier (GEM) detectors in the magnetic field generated by the Goliath magnet achieved three main results: • reproducing the mechanism of generation of short circuits observed in the GE1/1 detectors installed in CMS, when a magnetic field ramp is performed with the High Voltage (HV) OFF.The test provided a probability of short circuit generation per magnet ramp  HV OFF short = 0.42 +0.94 −0.35 %, at 68% confidence level (CL).In a complementary manner, the test also fixed an upper limit on the probability of short circuit generation with HV ON:  HV ON short < 0.49% (at 68% CL).
• setting an upper limit on the discharge probability per HV transition from the STANDBY to READY_FOR_PHYSICS configurations: as described in section 3.3, no discharge was observed both at 68% CL in the GEM mode (< 5.57%) and the CMS experimental site mode (< 5.93%).
• providing an interpretation of the discharges observed during the magnetic field ramps: the evolution in time of the discharge rate per magnet ramp is not incompatible with the hypothesis of dust trapped inside of the detector.It is not possible to verify the origin of the dust in this test, and it is hypothesized that it could be constituted of residuals trapped in the detector during the assembly and installation phases, or a result of discharge events damaging the detector and thus producing residuals of dust.
The dust hypothesis fits also with the increase of the discharge rate, in case of mechanical stress applied to the detectors or in case of the first inversion of the magnetic field direction.
The results of the test had an impact on the operation of GE1/1 detectors installed in CMS, by defining a safe procedure for magnet ramps.The adopted procedure consists of: • keeping HV ON for power supply channels feeding GEM foils and OFF for those powering the gas gaps; • increasing the limits for HV trip in the A1515 boards:  0 = 20 µA and trip time equal to 3 seconds.
A further investigation on the nature of particles trapped inside the detectors, leading to discharges, will be possible only when the access to the detectors installed in CMS will be granted for an extended period of time.This study anyway exploited the discharge and short generation probability for the specific detectors used during the test, in quantitative terms and as point of comparison for future studies.

Figure 1 .
Figure 1.Schematic drawing (same comment for other captions) of the installed GE1/1 detectors in staggered arrangement in the CMS experiment [3].Highlighted in blue (light) are the long detectors and in red (dark) are the short ones.Reproduced from [3].CC BY 4.0.

Figure 2 .
Figure 2. Diagram showing the functioning principle of a triple-GEM detector: an avalanche induced by a muon crossing the detector develops from drift, across the three foils, and generates the signal on the readout.

Figure 3 .
Figure 3. Scheme of the HV circuit used to power each GEM foil in a GE1/1 detector.

Figure 4 .
Figure 4. High Voltage powering scheme of GE1/1 detectors.The naming conventions used for HV channels that are powered and monitored by the A1515 board are reported.The A1515 board applies a potential difference between two detectors planes and then monitors the current flowing in the power supply to maintain the potential difference applied.In addition, the naming conventions of electrodes when monitored by a 7-channel pico-ammeter is also reported.Differently from A1515 board, this device monitors the current flowing on each plane of the stack.

Figure 6 .
Figure 6.Magnetic field measured inside the CMS experiment at the nominal value [5].GE1/1 position is superimposed.Reproduced from [5].© 2010 IOP Publishing Ltd and SISSA.All rights reserved.

2 ; - 9 -•Figure 8 .
Figure 8.(a) Operating conditions at the moment of short circuit creation from the beginning of the GE1/1 commissioning up to the 5 th November 2021; (b) foils affected by short circuit from the beginning of the GE1/1 commissioning up to the same date.

Figure 9 .
Figure 9. (a) Visual representation of the magnetic field in the Goliath magnet in the XY-plane.In brown the triple-GEM detectors arrangement for the magnet test during the operation with the magnetic field orthogonal to the chamber plane [9] (horizontal configuration); (b) photo of the actual position of the detectors inside the test magnet.(c) Visual representation in the XY-plane: same conventions of figure 9a with the detectors in vertical configuration, with the magnetic field parallel to the chambers plane; (d) photo of the actual position of the detectors in the vertical configuration.

Figure 10 .
Figure 10.Example of monitored currents and voltages, measured with CAEN A1515 board, during a magnet field ramp that caused a discharge and a trip.The four plots show the same discharge event.(a) and (b) show the monitored voltages and currents across the three GEM foils, while (c) and (d) show the same across the GEM gaps.The measured value of the magnetic field is also displayed in each case (blue dots and vertical scale on the right).

Figure 11 .
Figure 11.Example of discharges observed through pico-ammeter: (a) monitored voltages across the three GEM foils, (b) monitored currents flowing on GEM top electrodes, (c) monitored voltages across the GEM gas gaps, (d) monitored currents flowing on drift and GEM bottom electrodes.

Figure 12 .
Figure12.Evolution of the discharge rate per magnet ramp per detector.The Run number is an incremental number used to identify a data taking time interval, when a single kind of ramp was performed: a single magnet ramp with HV ON, or multiple magnet ramps with HV OFF (identified with red vertical dashed lines accompanied by a red number indicating the count of multiple ramps).The main detector parameters varied during the test were the gas flow and the HV working point, showed in the plot bottom bands.Other situations indicated on the plot are the first inversion of the magnetic field sign (red vertical solid line), the mechanical stresses (black vertical solid line) and the change in the detector orientation (figure9c) respect to the  field direction (yellow box in the right part inside the plot).During all the previous test period the detectors' planes where orthogonal to the  field direction (figure9a).

Table 2 .
Thickness of the gas gaps used for triple-GEM detectors in the GE1/1 station.

Table 3 .
Potential difference applied across the GEM foils and gas gaps for the set equivalent divider current   .This is not the absolute voltage applied to the electrode, referred to the ground.

Table 4 .
Electric field generated inside the GEM foils and gas gaps for the set equivalent divider current   .