1 Introduction

Worldwide business trips are strictly restricted due to the COVID-19 pandemic. The number of passengers via airline traffic is decreased by half compared to the ones before the pandemic as shown in the Fig. 1. As a consequence, research activities in the area of the high-energy physics have been significantly affected including on-site detector maintenance and data-taking.

Fig. 1
figure 1

The world passenger traffic evolution by airplane including the COVID-19 period [1]

Full size image

The e-Science paradigm of particle physics has been suggested to overcome physical distances and time zone differences between researchers and the experimental sites [2,3,4]. It provides an environment wherein on/offline activities can be performed regardless of the physical location of individuals, therefore allowing continuing experiments without any interruption [5]. We have constructed remote control rooms at the Korea Institute of Science and Technology Information (KISTI) for the Collider Detector Fermilab (CDF) experiment and the Belle-II experiment as shown in Fig. 2. In this article, we introduce our experience in constructing and operating remote control rooms, including international networks, security, data acquisition (DAQ) systems, data transfers, and monitoring systems [6].

Fig. 2
figure 2

KISTI remote control room for CDF experiments at Fermilab and Belle II experiments at KEK

Full size image

2 KISTI remote control room

Modern high-energy physics experiments use enormous amounts of resources for data taking. Detectors consist of multiple sub-detectors built with various principles with millions of electronic channels running in different conditions. In the experiments at the high-energy particle accelerators, the DAQ system has to be fast to enable data-taking decisions in a real-time. To ensure the entire system of the DAQ is reliable, a team of shifts with well-defined procedures to perform continuous and reliable data-taking is necessary.

There are several types of shifts for particle collider experimental physics: (1) on-site shift that can be pursued at the accelerator laboratory, (2) off-site shift that can be pursued outside of the accelerator laboratory, (3) online shift to acquire data from particle collider experiments, (4) offline shift to process the data using computing resources, and (5) remote shift that takes a shift at off-site as if at on-site. Table 1 presents the remote shifts in both CDF and Belle II experiments. Table 2 shows the comparison of online and offline shifts between CDF and the Belle II experiments.

Table 1 Remote shift types and roles of CDF and Belle II experiments
Full size table
Table 2 The comparison of online and offline shifts between CDF and the Belle II experiments
Full size table

KISTI have been establishing remote control room under the e-Science paradigm, starting from 2008. As shown in the Fig. 3, the first online remote shift was performed successfully from Aug. 1 of 2008, for 4 years until the closing of the CDF experiment. The main contribution of the shifts performed at the KISTI remote control room was the DAQ of the CDF experiment by performing a shift of 80 days for 4 years.

Fig. 3
figure 3

The period of operation of KISTI remote control room

Full size image

In 2013, the remote control room at KISTI was upgraded to cover the offline shifts of the Belle II experiment at KEK from Mar. 11, 2013. To operate online remote shifts during the COVID-19, there was an upgrade of the control room in Feb. 16, 2021, during the COVID 19 pandemic.

Reliable network connections among KISTI in Korea, Fermilab in the USA, and KEK in Japan are required for a smooth remote shift. The international network is illustrated in Fig. 4. For the CDF experiment, the connection between KISTI and Fermilab had been prepared from the 1 Gbps GLORIADFootnote 1 (GLObal RIng network for Advanced application Development) since 2005. Now, two international networks are available at 100 Gbps which allows fast and reliable connections between KISTI in Daejeon, Korea and Fermilab in Chicago, USA. The primary network connection is established through the Korea Research Environment Open Network 2 (KREONet2; the Korean gigabit network for research) [9] between KISTI through Seattle to the gateway in Chicago. The Chicago gateway is connected to Fermilab. The other KREONet2 line is connected directly between Daejeon and Chicago. These two networks of KREONet2 connect the Fermilab and KISTI remote control rooms stably and provide a stable environment for remote shifts.

Fig. 4
figure 4

Schematic of international network among Fermilab, KEK and KISTI

Full size image

For the Belle II experiment, KREONet2 connects the KISTI gateway and KREONet2 Hong Kong gateway stably at a speed of 100 Gbps. The Hong Kong gateway of KREONet2 is connected to the Hong Kong gateway of Japan Gigabit Network-X (JGN-X; a high-speed research network in Japan) to provide a stable network to Tokyo at a speed of 100 Gbps [10]. The Science Information NETwork 4 (SINET4) research network connects the Tokyo gateway and KEK gateway with a speed of 100 Gbps [11]. The backup high-speed network through Seattle has been incorporated when the network to Hong Kong encounters connection problems. Global Research and Education (R&E) network is also established as shown in Fig. 4.

3 CDF remote control system

3.1 CDF experiment

The CDF experiment was an accelerator-based high-energy beam collision experiment using the Tevatron accelerator at Fermilab, USA [12]. The CDF experiment collided protons and anti-protons with energy of 0.98 TeV [12]. These experiment played a significant role to search for the Standard Model and the fundamentals of nature, such as the discovery of the top quark. This was a large collaboration consisting of 625 collaborators from 15 countries including the USA, France, Italy, Japan, Germany, England, and Korea [13]. The Korean CDF group consisted of approximately 30 individuals affiliated with KISTI, Kyungpook National University, Seoul National University, Sungkyunkwan University, Jeonbook National University, and Jeonnam National University [13]. The CDF Run II experiment began in 2001, and the beam collision data were collected until the end of Run II in 2011, as shown in Fig. 5 [13]. Both online and offline shifts were performed simultaneously in the CDF experiment. A remote control room was used for the both online and offline shifts. Those were duties to be performed as a member of the CDF experiment. The Korean consumer operator (CO) could perform an online shift through the remote control room for eight hours per day without going to Fermilab, USA. The CO shift worked on the monitoring and control of various types of applications for eight hours.

Fig. 5
figure 5

The integrated beam luminosity of CDF experiments and period of remote shift with KISTI remote shift room [13]

Full size image

3.2 Online remote control system

Figure 6 introduces the communication between the CDF main control room at Fermilab and the CDF remote control room at KISTI. It is connected with Polycom telecommunication devices. Three shifters with different roles were assigned as a group in the CDF experiment. The first was the Science Coordinator (SciCo), who had extensive experience in high-energy particle physics and was responsible for the shift session [13]. The second was the Ace shift, who was an expert on the components of all detectors and control of electronic reading devices [13]. The third was the CO, who uses the remote control panel to control the experiment. The CO also monitors the status of DAQ, its transfer, and storage with data quality monitoring. Other duties of CO are recording the data and checking the data quality [13]. The CO shift could be operated off-site with certain requirements such as shift training, Polycom connection, and approval of the sites from the CDF management. To solve any problems in case of emergencies that can be occurred during the remote shift, the backup shift was assigned at Fermilab.

Fig. 6
figure 6

The communication between the CDF main control room at Fermilab and the CDF remote control room at KISTI [13]

Full size image

Among remote control systems, the Linux server ‘cdfroclnx1.kisti.re.kr’ has a security policy identical to that of the Fermilab in the USA. After the Linux server ‘cdfroclnx1.kisti.re.kr’ was Kerberized, the next step was to enable port-forwarding from the main server of Fermilab, ‘b0control.fnal.gov’ [13]. Due to the security policies of the U.S. Department of Energy and Fermilab, execution commands cannot be executed while being logged into the CDF online cluster from the outside [13]. Therefore, the X application called “concon (Consumer Controller)” should be sent from the on-site server to the remote control system at KISTI via the Kerberized secure channel. The port-forwarding provided by a secure shell was used in this system as shown in Fig. 7. If “concon” is sent to the remote control system, the remote shift can control whether viewed through “concon”. The remote shift performer check the consumer status through a web page or “concon” [13]. Figure 8 shows a view of the CDF main control room at Fermilab and CDF remote control room at KISTI [13].

Fig. 7
figure 7

The port-forwarding system for the secure system [13]

Full size image
Fig. 8
figure 8

The view of the CDF main control room at Fermilab (left) and CDF remote control room at KISTI (right) [13]

Full size image

3.3 Offline remote control system

KISTI performed another type of shift called remote data processing. The remote control room implemented the online function of remote data processing, whereas the offline function was implemented simultaneously. This offline shift is called the SAM data-processing shift.

Figure 9 shows the homepage of the CDF SAM DH system [13]. It was performed for 8 h per day for 7 days. Shifters at the off-site could perform their duties during the daytime in their time zone. CDF SAM DH is called offline because the data processing in this case involves monitoring the data entering the tape from the SAM station.

Fig. 9
figure 9

CDF data handling page for offline remote shift [13]

Full size image

Offline data were transferred between the SAM station and the mass storage system (MSS). In Fermilab, the mass storage system consists of a dCache and enstore system [14]. The dCache software was developed in collaboration with Fermilab in the USA and Deutsches Elektronen-SYnchrotron (DESY) in Germany. The dCache was configured in the form of a disk that connects the user to the storage. The enstore system is a direct interface to access files stored in tape drives for end-users [15]. When a problem occurred, the end-user could know the issued SAM station and data-farm equipment. The role of the SAM shifter is to monitor the status of the data handling system at the CDF experiment including the SAM, MSS, Name and DB servers.

4 Belle II remote control system

4.1 Belle II experiment

The Belle II experiment is an accelerator-based high-energy beam collision experiment at the SuperKEKB accelerator in KEK, Japan. The main purpose of the SuperKEKB accelerator (an upgrade of the KEKB accelerator) is to search for new physics beyond the Standard Model. Twenty-seven countries with 126 institutes have been participating in the Belle II collaboration [16]. SuperKEKB would examine new physics by performing high-precision measurements of CP violations and rare decays that occur through heavy quarks and leptons on the GeV mass scale. The first beam collision of the SuperKEKB was performed and the beam collision data collected on Apr. 26, 2018 [17]. The structure of the SuperKEKB has been upgraded to obtain high instantaneous luminosity and collect further data. As shown in Fig. 10, the total luminosity was 400 fb−1 in Jun. 2022 [18]. The objective of Belle II is to acquire 50 ab−1 of total luminosity. Online and offline shifts were adopted in the Belle II experiment. The main purpose of an online shift was to monitor the DAQ system during the beam collision experiments, whereas that of an offline shift was to monitor data processing using computing resources.

Fig. 10
figure 10

The integrated beam luminosity of the Belle II experiment during the period of remote shifts [18]

Full size image

Figure 11 shows the schematics of the online and offline remote shift system of the Belle II experiment. The role of KISTI in the Belle II experiment is to develop and operate the data-handling system. ARDA Metadata Grid Application (AMGA; developed by the KISTI) is a software that manages data distributed in multiple repositories [19, 21]. The authors have contributed significantly to the adoption of AMGA in the Belle II experimental metadata service. In addition, KISTI operates the Belle II Grid Farm Tier-2 center, where Belle II data are transmitted and stored. A Belle II remote control room was constructed to monitor the data-handling system. A continuous offline shift was done after the first successful data production shift on Mar. 11, 2013.

Fig. 11
figure 11

The schematic of the online and offline remote shift system of the Belle II experiment

Full size image

It is difficult to visit KEK in Japan due to the COVID-19 pandemic. A remote control shift was introduced to cover the control room shift that could be performed generally in Japan. Therefore, the remote control room at KISTI was upgraded to enable a remote control shift. We successfully embedded an online control room on Feb. 16, 2021. The Belle II remote control room at KISTI was set to perform both online and offline shifts. Table 3 shows the comparison of online and offline shifts at Belle II. The Belle II remote control room at KISTI enables us to conduct particle collider experiments during the COVID-19 pandemic. It also saves travel time and costs.

Table 3 The comparison of online and offline shifts at Belle II experiment
Full size table

Table 4 shows the systems in the Belle II remote control room at KISTI. It consists of four servers and seven monitors for shift duty. There are two additional servers for the auxiliary system: a Polycom communication machine, network-attached storage, fax machine, printer, and telecom to support the shift. The system built in the KISTI remote control room enables a maximum of two individuals to perform each shift.

Table 4 The list of systems for KISTI remote control room
Full size table

4.2 Online remote control system

Figure 12 shows the communication between the Belle II main control room at KEK and Belle II remote control room at KISTI. The communications between the on-site and remote shifters are done using SpeakApp and Rocket.Chat. The Belle II online shift system consists of an on-site shifter and two remote shifters. The on-site shifter controls the beam collision in the Belle II control room. Meanwhile, the remote shifter assists the on-site shifter, monitors the DAQ system and records the shift logs.

Fig. 12
figure 12

The communication between the Belle II main control room at KEK and Belle II remote control room at KISTI

Full size image

Figure 13 shows how to connect the monitoring pages for online shifts. The remote shifter connects to the remote control panel and high-voltage panel using the KEK virtual private network (VPN) in the Belle II remote control room at KISTI. The KEK VPN provides a secure environment for remote shifters. The on-site shifter uses the remote control panel to control the experiment. The remote shifter in conjunction with the on-site shifter controls the beam collision. The remote shifter also monitors the status of DAQ, its transfer, and storage with data quality monitoring of the DESY laboratory and the Grafana [22] data transfer monitor system of KEK. Other duties of remote shifters are recording the data log at the e-log and checking the data quality at the run registry recording page. There are four servers for remote shifts in the Belle II remote control room at KISTI. Therefore, two shifters may smoothly adopt each remote shift. Figure 14 shows a view of the Belle II main control room at KEK and Belle II remote control room at KISTI.

Fig. 13
figure 13

The schematic how to display the monitoring pages for online shift

Full size image
Fig. 14
figure 14

The view of the Belle II main control room at KEK (left) and Belle II remote control room at KISTI (right)

Full size image

4.3 Offline remote control system

An offline shift is also called a data production shift. Belle II Monte-Carlo simulation samples were generated at the Belle II Grid system. The Belle II data-handling system transfers and stores the generated MC data at the Grid sites. The main purpose of the offline shift is to monitor the Belle II data-handling system. In case of problems, shifters are asked to contact experts or site managers. KISTI has established the Belle II data-handling system and operated the KISTI Belle II Grid farm. We adopted an offline shift to monitor the status of the Belle II data-handling system and Grid farms.

As shown in Fig. 11, the offline shifter connects to the monitoring system via a Grid Belle virtual organization (VO) for security. Each monitoring system for offline shifts requires Belle VO for the connection. To consider this, each shifter has to issue the Grid Certification Authority (CA) from their authorized institute. KISTI contributes to the security system of offline shifts as an authorized Korean institute of Grid CA, called KISTI CA. The monitoring page assumes the shifters connect with the KISTI CA.

Figure 15 shows how the offline system displays the monitoring pages. The offline shifter connects to the DIRAC server to monitor the connection and job running status of each Grid farm site. The shifter can monitor the KEK server status through the connection to the Ganglia server. The Grafana [22] server provides efficiency meters of file transfer and deletion for each Grid farm site. The shifter examines and records the monitoring results on the e-log page every two hours. When a problem occurs, a shifter issue JIRA [8] ticket to report the problem to the Belle II distributed computing team to solve it.

Fig. 15
figure 15

The system how to display the monitoring pages for offline shift. The specification of the system is presented in Table 4

Full size image

5 Results

During the CDF II experiment (Jun. 30, 2001 − Sep. 30, 2011), the CDF remote control room at KISTI was operated from Aug. 1, 2008 to Sep. 30, 2011 [13]. Figure 16 shows the annual and total shift days of CDF online remote shifts at the CDF remote control room at KISTI. The error rate of a remote control room can be defined as [13]

Fig. 16
figure 16

a Annual shift days of CDF online shifts at the CDF remote control room at KISTI and b cumulative online shift days at the CDF remote control room at KISTI from 2008 to 2011

Full size image
$$Error \; rate=\frac{shift \; time \; operated \; by \; backup \; shifter}{shift \; time \; scheduled \; for \; remote\; shifter} .$$

The error rate of the CDF remote control room at KISTI was zero because the Fermilab backup shift was not operated during the period of the remote shifts. This implies that the CDF remote control room at KISTI provided a stable environment for shift duty. Figure 5 shows that the DAQ efficiency defined as data acquired per delivered had remained stable since the CDF remote control room at KISTI was operated. Each CDF shift duty is pursued during a week, 8 h per day. Total 2% of the CO shifts were covered at the remote control room at KISTI as the CO shifts.

As shown in Fig. 17, at the Belle II remote control room at KISTI, we have been operating the Belle II offline and online shifts since Mar. 11, 2013 and Feb. 16, 2021, respectively. Similar to the CDF II experiment, the error rate of the Belle II remote control room at KISTI with the Belle II experiment was zero. As shown in Fig. 17, KISTI performed 177 days of cumulative both shifts (124 days for offline and 53 days for online) up to Jul. 31, 2022. KISTI has been attaining a 517% of achievement rateFootnote 2 (139 days for both shifts) since Belle II online remote shift is operated (Jan. 1, 2021 − Jul. 31, 2022). This is the first rank among the 126 institutions in the Belle II experiment.

Fig. 17
figure 17

a Annual shift days of Belle II online shifts at the Belle II remote control room at KISTI and b cumulative online shift days at the Belle II remote control room at KISTI from 2013 to 2022

Full size image

6 Conclusions

We constructed the CDF remote control room at KISTI to establish an e-Science research environment that enables us to study the particle collider experiment irrespective of time and location. We successfully operated an online remote shift for DAQ and an offline remote shift for data processing until the end of the CDF experiment on Sep. 30, 2011. After the shutdown of the CDF experiment, KISTI remote control room is changed to operate a Belle II offline shift from Mar. 11, 2013. Since the on-site shift at Belle II is restricted due to the COVID-19 pandemic, the remote control room at KISTI has been upgraded to operate the Belle II online shifts as well as offline remote shifts. The Belle II remote control room at KISTI enables us to conduct particle collider experiments during the COVID-19 pandemic. It also saves travel time and costs by performing remote control shifts without on-site visits. In the Belle II experiment, KISTI obtained the first rank of achievement rate per duty for both online and offline shifts among the 126 Belle II institutions since Belle II online remote shift is operated (Jan. 1, 2021 − Jul. 31, 2022). This remote control system would be used in future collider experiments, such as the International Linear Collider, Circular Electron Positron Collider, and Future Circular Collider.