Design and implementation of the AMIGA embedded system for data acquisition

The Auger Muon Infill Ground Array (AMIGA) is part of the AugerPrime upgrade of the Pierre Auger Observatory. It consists of particle counters buried 2.3 m underground next to the water-Cherenkov stations that form the 23.5 km$^2$ large infilled array. The reduced distance between detectors in this denser area allows the lowering of the energy threshold for primary cosmic ray reconstruction down to about 10$^{17}$ eV. At the depth of 2.3 m the electromagnetic component of cosmic ray showers is almost entirely absorbed so that the buried scintillators provide an independent and direct measurement of the air showers muon content. This work describes the design and implementation of the AMIGA embedded system, which provides centralized control, data acquisition and environment monitoring to its detectors. The presented system was firstly tested in the engineering array phase ended in 2017, and lately selected as the final design to be installed in all new detectors of the production phase. The system was proven to be robust and reliable and has worked in a stable manner since its first deployment.


Introduction
The Pierre Auger Observatory [1] was originally designed to study ultra-high-energy cosmic rays (UHECR) with primary particle energy above 3 × 10 18 eV. It is located in the Southern Hemisphere near the Andes mountains in the south-west part of the province of Mendoza, Argentina. It uses a hybrid detection technique employing both a surface detector array (SD) and a fluorescence detector (FD).
The SD of Auger is composed of an extensive array of about 1600 water Cherenkov detectors (WCD), separated by a 1500 m spacing and covering an area of 3000 km 2 . As above 5 × 10 19 eV, the flux is of around 1 particle per km 2 per century, the large area of the observatory allows to observe more than 30 high energy cosmic rays per year [1]. A detailed description of the SD of the Auger Observatory can be found in [2].
Each of the 1600 surface detector stations includes a 3.6 m diameter water tank containing a sealed liner with a reflective inner surface. The liner contains 12 000 litres of purified water. Cherenkov light produced by the passage of charged particles through the water is collected by three nine-inch photo-multiplier tubes (PMTs) that are symmetrically distributed at a distance of 1.20 m from the center of the tank and look downwards through windows of clear polyethylene into the water. The surface detector station is self-contained. A photovoltaic system provides an average of 10 W for the PMTs and the electronics system consisting of a processor, GPS receiver, radio transceiver and power controller.
The surface detector electronics records the PMT signals, makes local triggering decisions, sends time-stamps to the central data acquisition system for the global trigger building, and stores event data for retrieval when a global trigger condition is fulfilled. Due to the low available bandwith of 150 Bytes/s per station, they must operate semi-autonomously, performing calibrations and taking action in response to alarm conditions at the station level. The electronic system was designed 15 years ago using the technology available at that time.
AMIGA (Auger Muon Infill for the Ground Array) [3] is an enhancement of the Pierre Auger Observatory designed to satisfy two different objectives. The first objective is to lower the minimum detectable cosmic ray energy threshold to 1 × 10 17 eV by installing 61 SD stations with 750 m spacing in an infill array covering an area of 23.5 km 2 , which was completed in 2012. A more recent, and even denser array with 433 m spacing over 1.9 km 2 was also included and completed in 2019. Figure 1 shows a representation of AMIGA. The second objective is to enhance the capability of Auger to study ray composition by a direct measurement of the muonic shower component. The muon detection is achieved by 30 m 2 scintillator detectors buried 2.3 m underground alongside all surface stations. The 540 g/cm 2 of overburden determined by the local soil density completely shields the electromagnetic component of the showers and only muons of energy greater than 1 × 10 9 eV are capable of hitting the detectors.
The main function for the AMIGA Data Acquisition System is to manage the data transfer from the muon counters in a modular, flexible, easily configurable and scalable fashion compatible with the Pierre Auger Observatory data trigger. These characteristics led to specifications on how to design the firmware, software, hardware, network structure and protocol, and finally the monitoring system. For each of these important aspects of the data acquisition system, we dedicate a section on this paper, where we describe in more detail the motivation and the chosen solutions.
The AMIGA buried stations work in sync with the SD, receiving the local triggers by their accompanying WCD. The energy and geometry of the showers are reconstructed by the SD while extra information on the particle composition is provided by the muon-related observables such as the timing and size of the signal provided by buried scintillators.
In this paper we describe the features of the embedded system design for the AMIGA buried scintillator detectors, dubbed Underground Muon Detector (UMD). This description includes the electronics, as well as the synchronization with the associated SD station. As an illustration of the functionality of the whole acquisition chain, we conclude showing an event measured by the seven stations that compose the engineering array.
The main contribution of this work to distributed embedded systems in general is the following: We present in this paper a flexible scalable massive system of multiple distributed muon detectors that can be organized in a remote fashion, using existing tools for a very specific problem: the detection of the muon component of cosmic showers. In this paper we describe how we organize the detection system using the preexisting trigger conditions of the surface detectors in the Pierre Auger Observatory and how we can apply reliable existing technology that can be adapted and easily applied as a solution to this problem. The electronics system of the front-end is described in a companion paper [4] where the details of the front-end electronics of the UMDs, its design and performance are described. In summary, we must mention that the UMDs work as muon counters, using two types of detection: 1 bit multichannel digitalization, one for each of the 64 channels, and a 14 bit Analog to Digital Converter with two independent channels that measure the charge of the signal from the 64 channels for a muon count that exceeds 64 muons.

Underground Muon Detector overview
As the main objective of the UMD is to count the number of muons in a cosmic shower produced by a cosmic ray, the original design of AMIGA was based on particle detectors that would be able to count individual muons in a radiation detector. This detector was implemented as a segmented scintillator, divided into three modules of 64 channels each, with a total area of detection determined via simulations [5]. These simulations were also used to obtain the optimum detector segmentation (192 scintillator bars). The total area needed of 30 m 2 was therefore covered by three modules of 10 m 2 each.
All SD stations in the infilled array have been installed and are operating since 2008 (61 stations) [6]. The SD stations in the infilled 750 m array have been installed since 2008 (42 additional stations) [7] and the complete grid is operative since 2012. More recently, the SD 433m array was also completed. The companion UMD had an engineering array (EA) phase of seven stations to validate and optimize the detector design. This prototype phase ended in 2017 with two major changes: the replacement of the optical device from PMTs to SiPMs and improved electronics, which is presented in this work. Immediately after the EA phase, the production phase started and the UMD array completion is foreseen by the end of 2022 [7].
AMIGA uses a telecommunications system that consists of a point-to-multipoint 802.11n WiFi radio link, with redundant coordinators located at the Coihueco fluorescence detector building (about 6 km to 10 km link distance). That link provides TCP/IP remote access to the LAN of each station [8]. The system is also designed to be energy efficient (as power from the photovoltaic system is limited) and has the capacity to transmit a high amount of data from those remote areas. All AMIGA stations are expected to work at least 10 years in remote (and possibly hard to reach) areas. Under these conditions, the systems need to have a long service life and to allow remote access for diagnostics and early fault detection for high quality of data.
The electronics were designed as embedded systems to ease eventual hardware upgrades. They run Linux as open source operating system to be free from any particular commercial software and be able to implement different hardware choices used on the detectors.

Electronics Overview
The electronics for the UMD was designed to provide a reliable data transfer from the buried scintillator modules to the main data storage server in Malargüe. In order to do this, each UMD is connected to an SD to use the Pierre Auger Observatory trigger. In order to provide a safe data transfer and not to overload the existing SD telecommunications and power system, each station was equipped with a new telecommunications system [8], a new solar power system [9], a trigger relay system (the distributor, see section 2.5), three buried modules that comprise the UMD itself, and a synchronization system that connects the UMD data acquisition system with its corresponding SD data acquisition system. The motivation for the telecommunications system is stated in the mentioned reference, as a summary, an 802.11n WiFi system running a TCP/IP network is used. The motivation for the solar panel system design is stated in the mentioned reference. In summary, a 24 V power system was implemented to provide the power needed via two deep discharge 12 V batteries connected in series that satisfy the power requirements of the UMDs and their telecommunications system, a total of ∼46 W in the engineering array for at least one day without solar energy (For production the power of the buried electronics of the UMDs was decreased to ∼3.6 W as measured in the field, reducing the total power budget to ∼19 W per station in the main array, allowing the photovoltaic system to have an extended autonomy of at least three days without solar energy). The trigger distribution and synchronization mentioned was motivated by the need to provide muon counting data stored in the UMDs each time a trigger request from the main data processing center in Malargüe is received. Therefore, the UMDs must be synchronized with the SDs and the trigger lines must be distributed to each of the three buried muon counters in each station.
A drawing that illustrates the design of the SD and UMD combined detectors at a position in the infill is shown in figure 2 with its main components. Three 10 m 2 buried modules are used at each surface detector station for the final design [5]. The block diagram of the electronics is shown in figure 3, which describes the components of one combined station and its interconnection as detailed in the following subsections.

UMD scintillator module
In this section, we present a summary of the underground scintillator module and its electronics functionality. A thorough description of the scintillator module and the buried electronics has already been published in [10], [11] and [4]. In summary, the motivation for the design of this electronics was to provide a ring buffer that serves as a temporary storage big enough to save muon data of a full cosmic shower for the time the Auger Observatory trigger takes to send a request for that data. In this section we introduce the scintillator module, the hardware of the underground electronics, the structure of the stored data and the requirements for its implementation in the digital back-end of the underground electronics. We also review the hardware structure of the buried electronics interconnected with the scintillator bars using optical fibers.
Every module comprises 64 scintillation bars, each of 41 mm width, 10 mm thickness and 4 m length for a detection area of 10 m 2 . The light produced in each scintillator bar is collected and propagated along a wavelength-shifting (WLS) optical fiber of 1.2 mm diameter glued into a lengthwise groove of the bar. All fibers connect to an optical coupler (also called "cookie") located in front of the light sensor. The 64 scintillators and optical fibers are enclosed within a PVC (Polyvinyl Chloride) casing and they form together with the electronics the detector module.
A detailed description of the AMIGA detector is found in [10].
Two groups of 32 scintillator bars are mounted in each module at opposite sides of a central dome that contains the light sensor and electronics. For the final design we selected the Hamamatsu S13361 Silicon Photomultiplier (SiPM) [11]. An integrated electronics acquires the analog signals from this SiPM, processes them and provides control, monitoring and communication functionality to the module. The system records event data synchronized with the associated SD station (at about 100 events/s) and stores them for roughly the same time as the SD station. Each recorded event is composed of 2048 samples of 64 bits measured at 320 Msps and 1024 samples of two 14 bit ADCs (high and low gain) measured at 160 Msps [4]. The former and the latter are dubbed the binary and the integrator acquisition modes respectively. Then, an event with a temporal length of 6.4 µs requires: 64 bits x 2048 + 2 x 16 bits x 1024 = 163 Kbits. Notice that the 14 bits traces are sent in two Bytes, hence the 16 bits value in the second term. Added to this trace are a header of 336 bits, a trace of 2 x 2048 bits that correspond to the OR gates in the two CITROCs of the front-end [4] and 16 bits x 2048 that are the identifiers of each the bins in the 64 bits trace. All this adds up to a total number for the trace of: 163 Kbits + 336 bits + 2 x 2048 bits + 16 bits x 2048 = 201 Kbits. In order to store the data of a candidate event for about 20 s at a rate of 100 events/s, the minimum memory needed is 402 Mbits.
The electronics for each module consists of the front-end board and the Acquisition and Control board. The front-end processes the 64 light signals by two Application Specific Integrated Circuit (ASIC) [11] and two 160 Msps 14 bit ADCs (Analog Devices ADS4246) digitizing the sum of all SiPM channels (one with a low-gain amplifier and one with a high-gain amplifier). Each ASIC channel provides a pre-amplifier with programmable gain and a fast shaper with 15 ns peak time. Finally, a discriminator digitizes the signal into one bit. The discriminator threshold has a coarse setting via a 10 bit DAC (common for the 32 channels) and a per-channel fine setting using individual 4 bit DACs. The Acquisition and Control Board (also called in this paper the back-end) records the digital information of the front-end board and stores the data in a ring buffer when a trigger condition is found. Details are given in section 3.

Synchronization
In order to comply with the trigger structure of the Pierre Auger Observatory, we need to establish how the synchronization of the UMD will be carried out whenever a cosmic shower is detected by the SDs. The detailed description of the trigger structure of the observatory is already thoroughly described in [12] and [1]. In this section we outline the main restrictions our Data Acquisition System has to abide by in order to comply with these trigger requirements. The specifications described in this section serve as a reference for the trigger interconnection between the surface and buried electronics, the memory usage in the ring buffer and the data handling that has to be performed in the SD in order to successfully transfer muon data from the UMD to the main server in Malargüe that is effectively synchronized with the detection of a cosmic shower.
The Auger surface detector data acquisition and storage system is comprised of a central storage server (or "CDAS" located in the city of Malargüe) and several client stations; the muon counter replicates this structure, merging the muon counter data with the surface detector data in a post-processing step offline.
The trigger system of SD is hierarchical: The first level of trigger (T1) is generated independently by each station based on local analysis of the signals produced by its PMT. At calibrated SD stations the threshold is set for an average T1 trigger rate of about 100 events/s. These events are stored in a circular buffer with a capacity of 3000 events. Events are identified by a 64-bit timestamp (GTS) obtained from the GPS receiver with a latency of about 200 µs after T1 as measured in the laboratory. The station then applies predefined quality cuts to generate a second level trigger T2 with an average rate of 20 Hz. A list of T2 time-stamps is sent to CDAS once every second. CDAS searches through the list of T2 triggers for a coincidence in vicinity and time. When a coincidence is found, an array trigger (T3) is sent back to participating stations and its neighbors. Upon receiving a T3 trigger request, the station replies to the CDAS with the event data, if it is present in the circular buffer [12], [1].
UMD buried modules were designed to rely on an external trigger in order to acquire event data during a possible particle shower. To provide synchronization with the underground particle counter modules, additional electronics were installed in the surface station. These complementary electronics were designed to trigger and label the particle counters data when the surface detector encounters a T1 trigger condition, and to forward T3 data requests from CDAS [13] to the buried electronics afterwards.

Trigger output from Auger Surface Detectors
The time that precedes a T1 event strongly depends on the delay between the issuing of the trigger signal and the arrival of the identifier. To address this issue a buffer that stores the T1 data must take this delay into account. Also a faster additional time-stamp was implemented, capable of arriving at the buried modules within a few microseconds after the SD station detects the trigger condition. This time-stamp, called Local Time-Stamp (LTS) consists of a 24 bit number generated by the surface station. Taking into account the average T1 rate, each LTS value is guaranteed to be unique for at least 26.8 s, enough time to ensure that requested events will not have time-stamp collision (and therefore data collision) with newer events [14]. The time it takes between the detection of an event condition (T1) and the sending of the signal is about 760 ns. However, using this time-stamping mechanism requires a way of converting between the time-stamp used by the surface detector and the LTS. This conversion is provided by electronics named Trigger and Data Request (TDR). an auxiliar board retrofitted in every SD station with muon counter detectors installed ("Trigger-TX"). The minimum required data transmission rate is 10 Mbps (100 ns per bin). This board has two low-voltage differential signalling (LVDS) drivers. Each LVDS line uses a point-to-point configuration, using drivers and receivers from Fairchild Semiconductors (FIN1001 and FIN1002 respectively) supporting data rates higher than 600 Mbps.
The typical power consumption of the transmitter is 23 mW and of the receiver 13 mW, thus each data line requires 36 mW. This system uses two of the available four pairs of lines to send signals (data and clock), which gives a power consumption per link of 72 mW.
In order to avoid ground loops, the fast trigger signal is isolated before it reaches the LVDS driver using an Digital Isolator from Texas Instruments (ADuM1400), as shown in figure 4. The transmission line is terminated by a 100 Ω resistor placed very close to the receiver for best impedance match. An Unshielded Twisted Pair (UTP) Cat 5e cable with RJ45 connectors crimped at both ends (following TIA/EIA norm 586-A) is used to transmit the signal.
The  Figure 4 shows that the signal crosses two isolators, two LVDS transmitter / receiver pairs and two sections of UTP cables. With a 6 m cable between SD electronics and distributor and a 30 m cable to the UMD electronics the latency for signal transmission is 322 ns.
To calculate the total delay, the time needed for the acquisition system to recognize the trigger condition must be added to the signal latency calculated below. This time is about 500 ns therefore, the trigger of the AMIGA electronics will arrive about 1600 ns after the trigger signal has been generated in the Auger station electronics.

Trigger and Data Request
In the beginning of the engineering array phase the functionality of the TDR was realized by a Single Board Computer (SBC) with a buffered SPI receiver on the basis of an Altera MAX-II complex programmable logic device (CPLD). Since the production phase, an AMIGA Acquisition board is used for this purpose. With the installation of the new SD electronics called "Upgraded Unified Board" (UUB) [15] as part of the Auger upgrade, the TDR functions are available in this board and no auxiliary electronics are needed.
The TDR receives the GTS+LTS time-stamp from the Auger surface electronics via a 10 Mbps SPI channel and sends it to the SBC, which stores these time-stamps in a circular buffer with a depth of 2048 values. The TDR wiretaps the T3 requests to the station by a RS-232 interface at the moment. The UUB has all the hardware ports, software (T3 trigger to the buried electronics and distributor commanding/monitoring), and FPGA code needed to replace the TDR. Therefore, on stations with a UUB installed, the TDR and SPI line are removed, and the distributor control cable is connected to the UUB directly.
When a T3 is found, the TDR looks for the received GTS in the GTS+LTS table. If the GTS is found, its corresponding LTS is broadcasted to all the modules via UDP; otherwise, a "LTS not found" message is sent. The TDR also provides access to the SD serial console via SSH or TELNET protocols. Note that this trigger scheme has no dependency on the buried module electronics, and can be reused by any project designed to trigger with Auger surface detector stations.

Communications
The telecommunications system connects the AMIGA stations to the central storage server, and it comprises of an Ethernet LAN network (connecting the UMDs and the SD electronics) and a WLAN telecommunications system for the data transmission to CDAS. The WLAN is implemented as a point-to-multipoint WiFi radio link attached to the original Auger inter-FD microwave links and it is provided by a Mikrotik RB493 router, which includes an Ethernet switch and a 802.11n WiFi radio. These characteristics are mentioned in [8], where we can also find that the network structure for the UMDs was implemented over the 2.4 GHz WiFi band. The motivation for this decision was to have a modular system that is higly reliable to work outdoors and that can handle the throughput required for the network of detectors over the 2.4 GHz Wi-Fi band, for which the Pierre Auger Observatory has a licence for exclusive use in the local region granted by the Comisión Nacional de Comunicaciones of Argentina. The details of the design, the protocol as well as the tests performed to measure its reliability are described in the mentioned reference. In this section we expand the description of the network to the higher layers and how it is organized in subnetworks within each detector.

Network
The network addressing scheme for communications uses one /24 network (Class C network, 253 available IP addresses) for each station LAN, with all radios connected on the same WLAN network. (172.16. . /16). A station with an internal LAN network 1 .a.b. /24 will have the IP 172.16.a.b in the WLAN network. In the internal detector LANs, the IP 1 .a.b.1 is reserved for the gateway (WiFi radio), 1 .a.b.2 for the TDR, and 1 .a.b.(1 + x) for the module x of that detector (when x < 90).

Power and Trigger distribution
This section explains the distribution of the Pierre Auger Observatory triggers and their interconnection at a physical level with the UMDs. It is worth mentioning that the trigger and power distribution between a SD and the three buried UMDs are intimately related since they share the same lines, therefore we also include a brief description of the power and monitoring system used. As mentioned before, the main piece of hardware that is in charge of handling the trigger and power lines to the buried UMDs is called the distributor, and in this section we introduce its functionality.
The power for each station is supplied by independent photovoltaic systems since all installation sites are far away from any electric power grid [9]. The distributor receives power from the solar panels and distributes it to the four modules via Passive Power-over-Ethernet (PPoE) lines. It uses the same cable to distribute the fast trigger signal sent to the buried modules.
The distributor converts the LVDS signals from the Auger surface electronics to TTL-CMOS logic using LVDS receivers. The TTL fast trigger is isolated in order to avoid a possible ground loop and is split into four lines. Each line is independently converted to LVDS signals, available at the distributor output ports. Each PPoE output uses the pin-out of 802.3af mode B and is connected to the power supply through a relay that allows switching the power on and off. The relays are commanded by the TDR using RS-232 messages transmitted via UTP cable, allowing up to four distributors in a daisy-chain (required when more than four counters are present in a station).

Acquisition and Control System
In this section we present the functional structure of the Data Acquisition and Control (DAQ) system of the UMDs. We describe the implementation at the hardware level on a Cyclone IV FPGA, the timing issues and memory requirements for the tasks performed by the DAQ system, including the firmware of each UMD and the soft microprocessor platform used to implement every function required by the UMD. We also describe the power system distribution and consumption of the electronics, the interface required and finally a short description of the printed circuit board layout and how it was designed. As we will see in this section the constraints of speed and memory are significant, therefore we emphasize the need for a soft microprocessor platform in order to have easy access to upgrades or future hardware changes that may be required due to spares availability. In this way we have a flexible and scalable embedded system that can be used for any kind of particle or radiation detector that works synchronously. The Acquisition and Control System is composed by two distinct parts: A soft microprocessor and custom acquisition code (both detailed below). A soft microprocessor was chosen to simplify the electronics, reducing the component count and improving the electronics future-proofing.

Hardware
In this section we describe the back-end digital hardware used for data acquisition on each UMD. We introduce the basic structure of the main system, comprised of an FPGA and a memory that functions as a ring buffer storing data according to a trigger provided by the SD. The main CPU controls the interface and data transfer to the surface, as well as monitoring several key hardware parameters that indicate its good performance. As the main motivation for the design, we should mention the need for a compact flexible embedded system that can easily be reprogrammed remotely, as we will see in the following sections. As mentioned, the CPU and firmware were implemented using an FPGA connected to two physically independent LPDDR1 memories. This board has an expansion port with 150 pins using a 1.8 V single ended voltage standard for fast speed digital signals. Additionally, one RJ-45 connector receives the fast trigger (T1) and the clock line with LVDS receivers, another one uses a LVDS transmitter/receiver pair for serial communication. These LVDS drivers can be fed by power from the connected cable, or by power from the Acquisition board, selected by a jumper. All input/output signals from LVDS transmitters/receivers are electrically isolated. In the following subsections we describe the hardware used with its specifications in terms of timing, resources and memory usage, the printed circuit board design and the interface with the SD and the telecommunications system. 24 VDC on pins 4 and 5 and GND on pins 7 and 8

FPGA and memory
The board is built around an Altera Cyclone IV FPGA (model EP4CE40U19I7) which provides 39 600 logic cells and a memory of 1.16 Mbits. Table 2 summarizes the usage of these resources for the custom logic (Firmware) and the LEON3 softcore CPU. While only 18 % of the logic cells are not used, there is nearly half of the memory cells (48 %) left for future improvements.
The chosen industrial package with 329 I/O pins (of which 257 are used) works in a wide temperature range (−40 • C to +100 • C), which is adequate for the operating conditions of the electronics (buried underground). The FPGA series is optimized for low power consumption with a low core voltage of 1.2 V. The firmware incorporates the Altera Remote Update system [16] with a remote recovery option, which ensures additional safety in case of an upgrade failure.

Component
Logic  The board has two physically separated memories, one for the soft-core CPU and one for the acquisition firmware. This design avoids logic elements for the synchronized access to the two RAMs, thus speeding-up the acquisition firmware and saving resources. Both memories are Micron LPDDR (low power DDR, model MT46H32M16LFBF-5), with 1 GB of capacity and a working frequency of up to 200 MHz (Both memories are used at 100 MHz).

Power sources
Based on the estimates for the required ratings of the different devices, we designed a cascaded high-efficient power system as shown in figure 7.   Table 3. Power consumption for each of the outputs shown in figure 7. Table 3 show the nominal values of current and power consumption as measured in the laboratory for each of the outputs in figure 7.

Interface and Communications
The board provides two expansion connectors: one for data/logic signals and one for power supplies. The data expansion connector is a single high-speed connector, Samtec QSH-090-01-LDAK-TR, capable of transmitting signals with a maximum frequency of 9 GHz. It also has a central ground bar that ensures low resistance for the return currents through the connector. The connector has 180 lines, with a pitch between pins of 0.5 mm.
The power expansion connector provides the 24 V input power (directly from the PPoE connector) and the 3.3 V power supply. Additionally, two control lines provide the functionality to switch on or off the connected expansion board.
The Ethernet PHY driver is a DP83848IV from Texas Instruments, which is supported by the Leon 3 built-in Ethernet MAC and the Linux operating system. The MAC address of each board is divided in two parts: The most significant 11 nibbles (44 bits) are a firmware compile-time constant, and the lowest nibble (4 bits) can be set using four switches in the board.
For the monitoring of the system voltages an ADS7828 ADC from Texas Instruments was chosen. It has 8 multiplexed input channels, uses an internal reference voltage, and supports I2C communications for transferring monitoring data to the soft-core.

Printed circuit board
Based on the maximum working frequency within the board (∼200 MHz) it was decided that the appropriate material for this design is FR4, which can accommodate working frequencies of up to 500 MHz. A 12-layer design was made, following the recommendations of the FPGA and RAM manufacturers. The FPGA recommendations specify tracks with a minimum width of 5 mils and a characteristic impedance of 50 Ω. The impedance of the lines depends on the materials (dielectrics and conductors) and their geometry, that is, track width, thickness of the dielectric, among others. A standard thickness (1.6 mm) FR4 PCB with 12 layers does not allow 50 Ω lines, therefore a thicker (2.4 mm ) PCB was used.
Following the application notes of Altera, one LPDDR memory was connected to a whole FPGA I/O bank. The other memory was connected using part of a bank for the address pins and part of another for the data bus, as indicated by the FPGA manual [17]. Once this was defined, the routing of the tracks from the FPGA to the memories was done.
The tracks between RAM and FPGA must be striplines of controlled impedance and length to conserve signal integrity and the timing constraints imposed by the RAM. The track width and distance between the signal and reference layers needed for a specific impedance was determined by specialized programs taking into account the properties of the PCB material. They must also have equalised lengths to ensure that the propagation times of the signals are within the values recommended by the RAM manufacturer.

Soft microprocessor LEON3
The back-end design went through several iterations before arriving to the final design presented in this paper. Through this debugging process we concluded that the flexibility of the hardware was essential, as well as the remote control and configuration. After two prototypes were built, we chose to use a LEON3 soft microprocessor running at 50 MHz that has an open source architecture. It allows for the system to be fully programmable remotely from the Observatory campus in Malargüe, which eased its debugging on the field and allows for upgrades or changes in the firmware to accommodate future needs such as changes in the trigger system, calibration, monitoring and disabling malfunctioning parts of the hardware if necessary. LEON3 is a 32-bit CPU microprocessor core, based on the Scalable Processor Architecture Version 8 (SPARC-V8) instruction set architecture with an AMBA AHB system bus [18]. It was originally designed by the European Space Research and Technology Centre (ESTEC), part of the European Space Agency (ESA), and after that by Cobham Gaisler Research. It is described in synthesizable VHDL.
This Soft microprocessor has numerous advantages, particularly for a such long-running project: The base code (CPU without Floating Point support and most peripherals) is available under a free-software license (GPLv2), it is FPGA-independent (upstream code supports Altera, Xilinx and Microsemi FPGAs), comes with a complete library of peripherals, and allows a wide array of configuration options (both in adding and removing peripherals and selecting or removing CPU features to prioritize performance or resource usage). The peripherals used are: SPI Memory controller (used for early boot and remote Firmware upgrade), I2C (used for environmental sensors), RS-232 serial ports (used for High Voltage power supply and acquisition control/data transfer), GPIO ports (used for the front-end power supply control, reboot/power-off and watchdog), and 10/100 Ethernet MAC (used for communications).

UMD firmware
In this section we introduce the details of the firmware functionality for a UMD back-end. It handles the data flow and secondary tasks required of the UMD hardware. We describe each part of the firmware in a block diagram divided into three parts: the CPU core, the back-end main task of muon data storage and the interface with the front-end and the peripherals. The current FPGA implementation is divided into six main blocks, as shown in figure 8. The EMC block drives 128 MiB of external LPDDR1 memory for temporary event data storage. This data has a fixed size (2048 anode samples for each SiPM, plus data from both ADCs and the aux digital outputs from both ASICs, equivalent to 6.4 µs per event). The trigger block receives the T1 signal, immediately notifies the incoming event condition to the data block and forwards the LTS value corresponding to the incoming event once it has been fully received. The read-out stage processes the 64 digital inputs at 320 MHz and packs four samples into a single 256 bit word. To reduce the number of resources needed, in particularly interconnects, all firmware except for the readout process runs at 80 MHz. The data acquisition block stores 2048 samples of these 256 bit words in two alternating circular (internal) buffers. When a trigger arrives, the actual buffer is filled up and then switched to the alternative buffer. While the alternate buffer records the data, the recorded data is saved in the external RAM. The LTS table process manages a look-up table with 2048 entries. When addressed with an LTS of an event, the memory address of the event data block is returned, or an 'event not found' error is raised [14].
The ASIC programming block receives the ASIC configuration from the embedded system and then uses it to program both ASICs. The micro-controller interface block handles communications with the embedded system. This is implemented using serial ports. Finally, the coordinator block manages the rest of the blocks.

Remote upgrade
In order to have a flexible distributed system we need to provide a robust firmware upgrade system. Therefore we included in the main firmware a remote upgrade that can be performed by an operator and provide a failsafe if there is any problem during the upgrade. In this way we can remotely enhance and develop the main functions of the UMD. Altera Remote Upgrade works as follows: The hardware stores two FPGA images in a persistent memory, named "Factory" and "Application" in Altera literature. On FPGA power-on, the Factory image is programmed, which starts a watchdog and immediately programs the Application image. If the watchdog isn't periodically reset (at least once every 60 seconds), the Factory image is reprogrammed in the FPGA and takes over. Both firmwares can be identified by software via a GPIO pin. The Application image contains both a LEON3 and AMIGA code, while the Factory image contains only a LEON3. The watchdog is controlled by software, and the routines to reset it start late in the boot process. If the module boots in Factory mode, the software tries to re-enter Application mode ten minutes after booting has completed.
Firmware upgrades are done by writing a specially processed FPGA image into the SPI Flash. To prevent an overwriting of deployed software images by mistake, all Flash regions except the one containing the FPGA Application image are set write-protected.

Software overview
This section describes how the software in the UMD is implemented, the tasks provided to the UMD and the operating system used. The software is responsible for sending event data and monitoring data via a network to a remote location. To simplify and speed-up the development phase, this software runs over an operating system which also provides system tools. The system was designed to minimise the use of persistent storage in the module electronics, improving resiliency, easing electronics installation and replacement, and reducing the possibility of "bricking" the electronics on future software updates (the only persistent data on every module electronic is LEON3, Firmware, boot-loader and Ethernet MAC address). A description of the software, the boot process and the SDK follows.

Operating System
Following the open source philosophy adopted for the soft microprocessor, the operating system used is also open source. The underground scintillator detector modules and the TDRs use Linux [19] as the operating system. It was chosen for user familiarity, relative ease of use, and because it has full hardware support for LEON3 (CPU and required peripherals). Therefore the motivations for implementing the software in this manner is to facilitate remote upgrades and configuration, with a complete access to the hardware functionality of the UMD electronics deployed in the field. The current version used at the time of the writing of this paper is 4.9, with patches (provided by Cobham Gaisler) for hardware support forward-ported to 4.9. Additionally, the kernel was patched to use the built-in GPIO power-off driver, which has been-setup to instruct the FPGA to program the Application image. The system runs in RAM from an initcpio (embedded in the kernel image due to bootloader limitations). Loadable module support is disabled (all modules are compiled into the kernel image) because no out-of-tree kernel modules are used, the connected hardware is known in advance, and disabling module support results in smaller kernel and initcpio binaries.

Boot
Upon board power-on/CPU Reset, a boot-loader (uBoot 1.16 with Cobham Gaisler patches) starts running. This boot-loader brings up the Ethernet port, obtains an IP address via DHCP, and loads the full operating system image (specified by the DHCP reply) from a TFTP server located in the Mikrotik RB493 router in each station. The DHCP server runs on the central data storage server, with DHCP relays running on each station LAN. The TFTP servers are located in each station LAN to preserve WLAN bandwidth. This system allows for easy image upgrades: Simply overwrite the image on the respective TFTP servers and reboot the affected modules. Modules are assigned a fixed IP and hostname based on their MAC address. This address has the upper 44 bits set by the software image (common for all stations) and the lower 4 bits configurable by hardware-switches, admitting at most 16 module electronics per station. Since each station LAN is physically isolated and has its own network address, no MAC address conflicts arise as long as MAC addresses are not repeated within the same station LAN.

System Applications
All basic system applications (init, shell, etc) are provided by Busybox (currently v1.33.1) [20]. The most notable applications are: Telnet daemon, watchdog user-space application, NTP client, DHCP client and SPI flash. The DHCP client has been configured to obtain the client hostname published by the DHCP server (required by AMIGA software to identify the module), the station LSID (used by the monitoring software to identify the station), and the NTP server used to synchronize the internal time.

Software Development Kit (SDK)
The tool-chain used for software development (compiler, libc, linker and related utilities) was custom-built. It consists of a GCC 8.1 C compiler targeting by default the Leon3 architecture with software floating point support, using binutils 2.30 and uCLibc-ng 1.30 C library. The tool-chain was built for Ubuntu 18.04 running on Intel x86_64 compatible CPUs.

Software
In this section we describe in more detail the software structure and how it performs the tasks needed for the UMD to store data and transfer it to the CDAS. We organize this section into three main subsections that describe the most important routines and their functionality: Module configuration, data transfer, and calibration. The software was written to control the module and to manage event and monitoring data acquisition. It consists of five pairs of client-server applications. With the single exception of the Calibration software (MdCalib), all clients run automatically after booting in Application mode on each buried module electronics and all servers run automatically on boot on a server in CDAS, with the clients keeping a persistent connection to the server (unless specified). Additionally (and with the same exception), all programs use the host-name to identify each individual module, allowing the system to work behind a Network address translation (NAT). A brief description of each pair of programs, with the names of each program client/server side, follows.

Module configuration -MdCfg/MdCfgd
This software configures the front-end (CITIROCs, ADCs, and HV), the firmware parameters and provides configuration/status information (ex: acquisition mode, masked channels, etc) to other programs using a Unix socket. On start-up, the client connects to the server and requests the initial module configuration (stored in the central server in INI files). Once the configuration data has been sent and applied, this daemon disconnects from the server and keeps running in background, handling configuration and status requests from all other programs (these requests will be specified in each program's section).

Data transfer -MdSend/MdRcvd
This pair of programs handle the event data transfer and storage. When idle, the client periodically consults with MdCfg the current acquisition mode and the UDP broadcast port for data requests (only if in acquisition modes sending of data is requested). Upon a data request, the software instructs the firmware to search for the specified event, receiving either the event data or an error code, and sends the result to the server. The server keeps in RAM a dictionary of events indexed by the 16-bit internal CDAS Event ID. For each Event ID, the data for all participating counters is stored in a dictionary indexed by the counter ID. The server has a TTL counter for each event, which is decreased every 30 seconds and increased every time data for a new module arrives. Once this counter reaches zero, the event is written to disk and discarded from RAM (if more data for the same event arrives, it is treated as a new event and merged as part of the standard offline post-processing of AMIGA data). The event data is stored in JSON format, using a single file which is periodically restored and compressed (in gzip format) once per day. This format allows easy parsing using libraries available in most programming languages, and has a very good compression ratio when using gzip [21].

Calibration -MdCalib
These programs run the pre-defined Calibration routines on each module as described in [11]. On the module, MdCalib starts on boot and listens to calibration requests. Once a calibration request is received, the program acquires the file-lock for background data, sets the acquisition mode to 'Calibration' (all data requests are sent to the server indicating that the sent data is invalid due to the module being calibrated) and runs the requested calibration routine, setting the needed configuration parameters via MdCfg and sending the resulting background data to the requester. Once the calibration finishes, the module is returned to its previous state and the file-lock is released. On a remote machine, a dedicated program communicates with the calibration software running on one or more modules (identified by IP address) and receives the calibration data. Currently the request for calibration date must be issued manually by an operator, but full automation (via scheduled jobs) can be used.

Monitoring system
To guarantee the validity of the acquired data, the status of the detector, as well as the status of its measured data, have to be monitored. In this section we include a description of the monitoring system as implemented in the UMD and the main server. The motivation for the monitoring system is described below, in summary it is designed to perform two very important tasks: Background signal monitoring, where the record of the signal counts by the UMD is constantly taken, and environmental monitoring, where the hardware status (including power sources voltages, currents and temperature) is handled. The monitoring acquisition system was designed as a series of processes (one for each kind of monitoring) using client/server architectures, where each of the clients runs in a MC electronic and establishes a TCP connection to its corresponding server. The resources needed by the monitoring processes are extremely low, particularly from the client side: they use 1 MB of RAM that is activated every 5 minutes to send data to the server at the CDAS. The data volume sent is 1 KByte every 30 minutes. A description of these monitoring processes follows.

Background signal monitoring -backpulse_client/backpulse_server
These programs read and store the background pulse counters from the firmware. Background pulses are all the current pulses produced by the SiPM (not necessarily belonging to a possible event). The rate of these pulses for a given SiPM is expected to be relatively constant throughout the operating lifetime of the detector; variations on this rate over time may indicate aging or damage of any of the optical parts. The client periodically orders the firmware to snapshot the current counter values, which are then sent to the server and saved in space-separated files (one file per day and module), formatted with the module time-stamp and the 64 counter values. Since the background data client shares resources with the calibration daemon, this program tries to acquire a lock-file in a predetermined location before reading the counter data, and immediately closes this file after reading.

Environmental monitoring -monitoring_client/monitoring_server/data_handler
This trio of programs handle the monitoring data readout and storage. The client periodically reads the data directly from the monitoring ADCs and data from the HV power supply via MdCfg, and sends it to the monitoring server upon request. Monitoring data is taken every 5 minutes. The server spawns a new process for each connection request from a client which periodically requests its monitoring data. Every 30 minutes up to 432 Bytes of monitoring data plus a header of 35 Bytes is sent per station. This data is then immediately sent to the data_handler process via a unix socket. The data_handler process stores all the monitoring data it receives from its unix socket to a MySQL database and, as a backup, to a .csv file (rotated every 12 hours) that can be imported manually into the database.

Online-Monitoring
At the server side, the monitoring data is received and stored in a database. In this section we describe the monitoring data handling from all the UMDs deployed in the field and how it is presented to the user in order to visualize it and make decisions accordingly.  The main page of the monitoring system shows a Map option with the position of each SD station. Figure 10 shows an example, where the coloured marker depends on the alarm state of all the scintillator modules for each SD station. In the figure, green marks represent a station working in normal state, and gray marks represent stations that have yet to be installed.
Selecting a SD station shows the deploy view associated to this station and the monitoring data from the database for a particular date. An example of the monitoring system displaying a typical AMIGA station after deployment is shown in figure 11. Selecting a module shows the monitoring information and alarm state for each variable of the chosen module. This view also gives the option to mask alarms of any monitoring parameters and visualise the full monitoring history of the corresponding module.

AMIGA sample event
As an example of the acquisition process, this section presents data collected by the AMIGA UMDs for an UHECR of 4 × 10 18 eV impinging with a 20 • zenith angle , that hit the border of the muon counters. This event was recorded on May 9, 2018 at 14:44:56 UTC. The temperature measured in the SiPMs for the four modules belonging to station 1764 (which participated in the event) is shown in figure 12, taken form the monitoring system. Event data for each AMIGA station is shown in this section to illustrate the reconstruction procedure applied to the binary acquisition mode only .

Status of the AMIGA Array at the time of the event recording
As seven stations of the 750 m array were part of the prototyping phase (the so called Unitary Cell), the first operative hexagon for physics analysis has stations with higher segmentation (256 bars Values taken from the official Pierre Auger Observatory event reconstruction As the time of writing this paper, the reconstruction with the integrator mode is still under development instead of 192) for the projected 30 m 2 detection area as shown in figure 13. Two of these stations have four buried modules (two 10 m 2 modules and two 5 m 2 modules). One of these stations has eight modules (four 10 m 2 modules and four 5 m 2 modules) installed symmetrically with respect to the Auger Surface Detector; also known as a "Twin" (for validation purposes and to assess the resolution of the reconstruction procedure as described in [22]). One station has six modules (four 10 m 2 modules and two 5 m 2 modules), with the southern side of the station installed like the others, and the northern side with two 10 m 2 modules. The rest of the stations have one or two 10 m 2 modules.

UMD module binary data
AMIGA event data for each buried scintillator module is stored as lists of bin number/anode sample, only for samples where at least one channel has detected light (all other samples are not stored), and a list of bin number/sampled value for each ADC. The anode bin number indicates a time-stamp in 3.125 ns increments. These traces have a length of 2048 bins (6.4 µs). The length of the pre-trigger / post-trigger window is a configurable parameter of the electronics. In this example, the pre-trigger and post-trigger windows are about 4.4 µs/2 µs long, thus the trigger arrived at bin 1400.
Muon counts are obtained from these raw traces (shown as illustration in top panels of figure  14) in an offline analysis, searching for patterns in the signals of each channel using the technique described in [11]. A counting strategy must be applied to avoid undercounting from pile-up effects [5].
As an example of the reconstruction chain applied to extract a number of muons from the raw data, figure 14 shows the traces collected by two AMIGA modules at different distances from the shower core for the same event. It is apparent how the particle density and therefore the triggered raw traces decreases as the distance increases. In particular, the figure shows an extreme case for a very close-to-core detector at 145 m distance. In this case, the UMD is saturated as all 64 channels In the binary mode Applying the pile-up correction, this number of active channels translate into 33.7 reconstructed muons. This correction is applied to the number of active channels, and makes this number differ from the reconstructed muons. The method relies on a probabilistic model to estimate the number of particles, reporting the mean number of muons for a given number of input signals (or active channels).
As a final remark, it is important to stress that the detected particles arrived in a time window of about 200 ns and that there are no identified particles before or after this window.
The reconstructed number of muons for all AMIGA modules triggered in this sample event is presented in table 4. These counts were obtained using the Auger Offline software [23], which applies the already described counting strategy and the pile-up correction described in [5].
After the muon densities as a function of the distance are reconstructed, the observations are fit to obtain the estimator (450), the muon density at 450 m from the shower core. This observable is sensitive to the chemical composition of the primary cosmic ray entering the Earth atmosphere and is the one used for higher-level physical analysis. Combining this parameter with others from the shower reconstruction (as the energy, X max , etc), gives valuable information regarding the primary particle mass. Figure 15 shows the fit of the muon lateral distribution function made for the event studied following the procedure in [24].

Conclusions
The AMIGA front-end electronics was designed as an embedded system around the Cylone FPGA family. A soft-core LEON3 processor runs Linux as its operating system for high flexibility, short development time and other benefits. A front-end compatible with this board was designed and built for AMIGA buried muon counters using SiPMs as the sensing device and two CITIROC ASICs for signal conditioning. The system is capable of 64 SiPM channels sampled at frequencies of 320 Msps and two 14 bit ADCs (high and low gain) measured at 160 Msps. The first eight prototypes of this system were installed in October 2016. Afterwards, 53 electronics were built and are acquiring data since February 2018. After several tests, this design will be produced to complete the AMIGA muon counters, by installing three 10 m 2 modules in 61 stations.  [11] P A collaboration, Muon counting using silicon photomultipliers in the AMIGA detector of the Pierre Auger Observatory, Journal of Instrumentation 12 (2017) P03002. [16] Altera Corporation, Configuration and Remote System Upgrades in Cyclone IV Devices.