Execution data logs of a supercomputer workload over its extended lifetime

The data of this research describes the logged usage of the Euler cluster located at CIEMAT (Centre for Energy, Environment, and Technology Research), spanning the period between November 2008 and March 2018. The Euler database is open access in Parallel Workload Archive format, available from the PWA repository [1] and Mendeley Data [2], allowing in this way a whole new bunch of research possibilities on computer science.


Data
The whole execution trace of Euler cluster during its life is presented as raw data, which can be downloaded from our group's homepage [3], execution traces repository [1] and Mendeley Data [2] in the format of the Parallel Workload Archive [9]. The information contained is quite similar to the one in Euler logs, although curated and anonymized following the Spanish data privacy laws. It comprises 18 fields whose significance is specified in Ref.
[1] and replicated here for the sake of clarity and completion. The fields are the following: 1 Job Number (integer): a counter field, starting from 1. 2 Submit Time (in seconds): earliest time the log refers to is zero, and is the submittal time of the first job. The lines in the log are sorted by ascending submittal times, and the jobs are also numbered in this order. 3 Wait Time (in seconds): difference between the job's submit time and the time at which it actually began to run. 4 Run Time (in seconds): wall clock time the job was running (end time minus start time). 5 Number of Allocated Processors (integer). 6 Average CPU Time Used -both user and system (in seconds): average over all processors of the CPU time used, and may therefore be smaller than the wall clock runtime. If a log contains the total CPU time used by all the processors, it is divided by the number of allocated processors to calculate the average. Value of the Data Euler cluster database comprises a rich source of information regarding scientific users' behavior at computing. The statistical analysis of this database is helpful to the HPC administrators since it provides guidance about the real HPC needs. The data is helpful in better sizing the next HPC generation of supercomputers and to attain a high, steady resource occupation and the best revenue of the funding in terms of scientific results. Data is useful for system administrators and developers of HPC-related, mostly those involved in developing new algorithms for job scheduling, fault tolerance and minimization of energy consumption, just to mention a few. This data allow the HPC community to carry out new research on topics such as artificial intelligence methodologies applied to supercomputing, test of resource manager simulators for clusters, etc. This data are (up to the author's knowledge) the largest publicly available full workload dataset of a supercomputer already structured, making it easier for system users and developers to inspect the data. Fields 10, 16, 17 and 18 are unavailable from the PBS stored logs, thus a value of À1 is set throughout these rows according to the PWA format. Additionally, fields 13 and 14 are set to À1 when the owner of the job cannot be identified from the PBS logs and they always correspond to failing jobs. This is due to outages, hangs or reboots of the batch server or allocated nodes without job recovering, which usually imply a forced cleaning.
Some data statistics are plotted in Figs. 1e10. Fig. 1 shows the number of submitted jobs per year (submit time is added to the start time given in the description of data collection of the Specifications Table to distribute them per year). Two periods are visible in the plot: a first period with a growing tendency until 2012; and a following second period in which the number of jobs per year quickly decreases (cluster obsolescence). Fig. 2 shows the usage of CPU time over the years. Fig. 3 shows that a constant flow of jobs over time takes place in addition with bursts, that deeply affect the cluster occupation (i.e., about 10% of the jobs submitted in 2012 were submitted on a single day).
Euler is a highly parallel machine and Fig. 4 shows that besides its usage for parallel jobs, there is also a wide number of serial jobs executed. Next, Fig. 5 represents the average degree of parallelism over the years. There is a growing tendency of the degree of parallelism, driven by the increased maturity of the users and the adequacy of their codes to Euler capabilities.    The accumulative job submission by all the users except the 10 top ones has been included in Fig. 6 Besides, the number of jobs submitted by the most active users along the time is included in Fig. 7 (to increase legibility, in Fig. 7b the top 5 values have been removed). Both figures show that a small number of CIEMAT users employ the vast majority of resources: 150 users have submitted over 100 jobs, 90 users over 1000, and only 44 over 10,000 jobs. In fact, the top 5 users have submitted as many jobs as the rest of users all together using short periods instead of a regular, steady usage of the cluster. It can be seen that there is a constant flow of jobs by a large number of users, but the influence of the most active ones is overwhelming on some particular dates.
Walltime as a function of job size (number of MPI tasks) is shown in Fig. 8. Most of the parallel executions are powers of 2, as expected. The largest degree of parallelization is 512, which corresponds to a few of the total executed jobs (less than 100) over the cluster lifetime. Executions with a degree of parallelization of 256 have been performed on a normal basis.
Among the possibilities of the dataset exploitation, there is the statistical analysis to identify cluster misuses and bad praxis by some users, if any. Specifically, by means of the covariance operator it is feasible to determine those users which have been sharing over time, with a high probability, their cluster accounts to boost their computations (see Fig. 9, which plots a portion of the users, anonymized with IDs). This information is useful to the administrators in order to take actions to enforce the code of conduct regarding the access to shared resources at institutional level. In Fig. 9 it is shown that most  users exhibit a very small correlation coefficient signal (deep blue), but for some of them located out of the diagonal, the signal is significant (>0.6), which suggests a coordinated use of the accounts by one of the users.
A measure of the time that user's jobs wait for execution is provided by the slowdown metric, which serves to evaluate job schedulers when overload increases or special situations happens [11]. Slowdown is the job's response time (running plus waiting times) normalized by their running time. Utilization is the clusters' filling rate [12]. Both are built out of combinations of various fields of the dataset. The relationship between them is plotted in Fig. 10. The isolated circle at very low values of utilization corresponds to the initial tests performed during the commissioning of the cluster lifetime.    Queue 4 'batch': routing queue, which redirects the user request to a production queue depending on the requested walltime. Enabled on November 27th , 2009 Queue 6 'pruebas': tests queue, enabled on March 16th , 2009.

Benchmarking of
The differences among them lie in the available resources. The basic idea is that those jobs with less computational requirements be executed on the queues with higher priority. The scheduling policy is the so-called FairShare [10]. Basically, it prioritizes short jobs over long ones, serial over parallel ones, and the jobs of the less active users over more active ones. The timeframe for this is 24h. The set of Figures has been built by postprocessing the execution trace with Python and Matplotlib [5,6] on the open source database server MariaDB.
For the better interpretation of the dataset, it must be clarified that the community of users of cluster Euler is formed by scientific and technical personnel who has a contractual link with the institution. Exceptionally, some users are external to the institution but have been permitted to access to this HPC resource because they are participants in several CIEMAT projects.