Data for a meta-analysis of the adaptive layer in adaptive large neighborhood search

Meta-analysis, a systematic statistical examination that combines the results of several independent studies, has the potential of obtaining problem- and implementation-independent knowledge and understanding of metaheuristic algorithms, but has not yet been applied in the domain of operations research. To illustrate the procedure, we carried out a meta-analysis of the adaptive layer in adaptive large neighborhood search (ALNS). Although ALNS has been widely used to solve a broad range of problems, it has not yet been established whether or not adaptiveness actually contributes to the performance of an ALNS algorithm. A total of 134 studies were identified through Google Scholar or personal e-mail correspondence with researchers in the domain, 63 of which fit a set of predefined eligibility criteria. The results for 25 different implementations of ALNS solving a variety of problems were collected and analyzed using a random effects model. This dataset contains a detailed comparison of ALNS with the non-adaptive variant per study and per instance, together with the meta-analysis summary results. The data enable to replicate the analysis, to evaluate the algorithms using other metrics, to revisit the importance of ALNS adaptive layer if results from more studies become available, or to simply consult the ready-to-use formulas in the summary file to carry out a meta-analysis of any research question. The individual studies, the meta-analysis and its results are described and interpreted in detail in Renata Turkeš, Kenneth Sörensen, Lars Magnus Hvattum, Meta-analysis of Metaheuristics: Quantifying the Effect of Adaptiveness in Adaptive Large Neighborhood Search, in the European Journal of Operational Research.


a b s t r a c t
Meta-analysis, a systematic statistical examination that combines the results of several independent studies, has the potential of obtaining problem-and implementationindependent knowledge and understanding of metaheuristic algorithms, but has not yet been applied in the domain of operations research. To illustrate the procedure, we carried out a meta-analysis of the adaptive layer in adaptive large neighborhood search (ALNS). Although ALNS has been widely used to solve a broad range of problems, it has not yet been established whether or not adaptiveness actually contributes to the performance of an ALNS algorithm. A total of 134 studies were identified through Google Scholar or personal email correspondence with researchers in the domain, 63 of which fit a set of predefined eligibility criteria. The results for 25 different implementations of ALNS solving a variety of problems were collected and analyzed using a random effects model. This dataset contains a detailed comparison of ALNS with the non-adaptive variant per study and per instance, together with the meta-analysis summary results. The data enable to replicate the analysis, to evaluate the algorithms using other metrics, to revisit the importance of ALNS adaptive layer if results from more studies become available, or to simply consult the ready-to-use formulas in the summary file to carry out a meta-analysis of any research question. The individual studies, the meta-analysis and its results are described and interpreted in detail in Renata Table   Subject  Management Science and Operations Research  Specific subject area  Analysis of metaheuristic algorithms  Type of data  Table  How data were acquired For each individual study included in the meta-analysis, the ALNS and its non-adaptive variant ( ¬ A)LNS were run a number of times on a number of problem instances. These results from the individual studies were pre-processed and then analyzed with a random-effects model. Data format Raw Filtered Analyzed Parameters for data collection We performed a literature review of ALNS, restricting our search to articles that: • describe the weight adjustment mechanism used in sufficient detail, • employ a weight adjustment formula which includes a parameter, that could be set to a certain value so that the adaptive layer is switched off, and • employ a roulette wheel mechanism to choose between heuristics. Description of data collection The results of the comparison of ALNS and ( ¬ A)LNS for [1,2]

Value of the Data
• Detailed per-instance comparison results of ALNS with its non-adaptive variant across a number of independent studies, i.e., implementations of ALNS to solve a broad range of different problems, helps to evaluate the importance of ALNS adaptive layer. • The data are of greatest interest for researchers interested in Adaptive Large Neighborhood Search, and in particular its adaptive layer. Furthermore, the ready-to-use sheet with random effects model formulas can benefit those interested in carrying out a meta-analysis of any research question within operations research, or any domain. • The data enable to replicate the analysis, to evaluate the algorithms using other metrics, to study the influence of different factors on the added value of ALNS adaptive layer, to revisit its importance if results from more studies become available, or to simply consult the summary file for a meta-analysis of any research question.

Data Description
In adaptive large neighborhood search (ALNS), a solution is iteratively destroyed and repaired through the application of several heuristics h ∈ H. In order to select the heuristic to use, a weight is assigned to each destroy heuristic h ∈ D and each repair heuristic h ∈ R . First, weights are set to some initial values, which are usually equal. An adaptive weight adjustment procedure updates these weights based on the performance of each heuristic. At the end of each segment s (a number of iterations), the weight w s +1 h of the heuristic h is calculated as follows: where π h is the score of heuristic which reflects its performance, θ h is the number of times heuristic h was used during the last segment, and r is the reaction factor. The reaction factor r controls how quickly the weight adjustment procedure reacts to changes in the effectiveness of the heuristic, and if r = 0 , the weights remain unchanged.
This dataset contains a detailed comparison of ALNS and its non-adaptive variant ( ¬ A)LNS, for ALNS implementations described in [1,2,[4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22] . In other words, it lists the results of comparing ALNS with the value of the reaction factor r as chosen in each of the articles, and ALNS with r = 0 (without adaptiveness). The results from the individual studies are then processed, and analyzed with a random effects model. The dataset is structured into two folders and a summary file: The folder data_individiual_studies_raw.zip consists of the comparison of ALNS with the non-adaptive variant per study and per instance, in the format (.xslx,.xsl,.csv,.ods,.xml or.html) summarized by the authors of the individual studies and e-mailed to Renata Turkeš, and is only added for the purpose of completeness and transparency.
The folder data_individiual_studies_filtered.zip consists of the comparison of ALNS with the non-adaptive variant per study and per instance, and corresponds to the raw data from data_individiual_studies_raw.zip , but pre-processed by Renata Turkeš. Each of the files corresponding to an individual study S k is specified in the same format, the redundant data are removed, and some further information is calculated in order to summarize the impact of the ALNS adaptive layer for each study.
More precisely, a file corresponding to study S k starts with the information about the article title and objective function f k , with the main information summarized in a table. The first three table columns list instance names, and the average objective function value across a number of runs of the best solution found by ALNS and its non-adaptive variant, and are obtained from data_individiual_studies_raw.zip . These objective function values are then used to calculate the next four columns, which evaluate the improvement in the objective function value with the adaptive layer, and whether ALNS outperforms ( ¬ A)LNS or not. Finally, from this table we calculate some summary values for the considered study: average, variance and 95% confidence interval for the added value of the ALNS adaptive layer, across problem instances. The calculation of the improvement A k in the objective function value with adaptiveness in study S k , and the within-study variance V k is described in great detail in the next section on the experimental design. For example, the file grimault2017adaptive.xlsx corresponding to the ALNS introduced in [7] is summarized in Table 1 . The study weights W k are calculated using the within-study variance V k and the variance T 2 across studies. Table 2 lists the features that are calculated for each study, as described in detail in the next section on the experimental design. Table 1 Example of a file in data_individiual_studies_filtered.zip , summarizing the importance of the adaptive layer for a single individual study S k . improvement A in objective function value with adaptiveness, averaged across 10 runs,

Experimental Design, Materials and Methods
The summary effect A reflecting the importance of the adaptive layer is the weighted average Let us assume study S k considers a maximization problem with the objective function f k , } denote the set of available problem instances. We run ALNS introduced in study S k , with the value of the reaction factor r k chosen in the individual article, to find the solution x * r k (I) for problem instance I ∈ I k . The best solution found by the non-adaptive ( ¬ A)LNS with r = 0 for the same problem instance is denoted with x * 0 (I) . Since ALNS is not a deterministic algorithm, we run both algorithms several times on each problem instance, and calculate the average objective function values across a number of runs, f k (x * r k (I)) and f k (x * 0 (I)) .
The added value of adaptiveness in study S k is calculated as the improvement in the average objective function value yielded with the adaptive layer, across the set of available instances: If we are considering a minimization problem, the average improvement in the objective function for study S k is calculated as The weight W k of study S k is calculated as inverse variance. In a random effects model, variance is calculated as the sum of within-study variance and variance across studies. The withinstudy variance is estimated with the squared standard error: ( σ k is the standard deviation, and N k is the number of problem instances in study S k ). It makes sense to weigh studies with the inverse variance: we assign more weight to the studies which include a greater number of instances, and for which the dispersion of the effect size across instances is small. The between-study variance is estimated using the DerSimonian and Laird method: where: • Q is a sum of squares of the effect size estimates about their mean, weighted by the inverse of variance V k , • df degrees of freedom, df = K − 1 , where K is the number of studies included in the metaanalysis, • C is simply a factor which puts the standardized variation between studies Q − df back into the same metric that had been used to report the within-study variance, If T 2 is less than zero, it is set to zero, since variance cannot be negative. The total variance under the random effects model is therefore V k + T 2 , so that the weight of study S k is calculated as: In the remainder of this section, we illustrate the experimental design (i.e., all the calculations carried out in data_individiual_studies_filtered.zip and data_analyzed.xls ) with a small example of a meta-analysis with only two studies, with two and three considered problem instances, and two runs of the algorithms for each instance, summarized in Table 3 . The information in the highlighted upper-left rectangle corresponds to data collected from a single study S k : the objective function value of the best solution found by ALNS and the non-adaptive variant ( ¬ A)LNS, for a number of problem instances and algorithmic runs. This information is used to estimate the mean importance A k of the adaptive layer, i.e., the average improvement of ALNS upon the non-adaptive algorithm, for a study S k .
The weights W k of the studies are then calculated as the sum of within-study variance V k (square of standard error, which incorporates both the standard deviation σ k across problem instances within a study, and a number of instances N k ) and between-study variance T 2 (estimated with poor precision when the number studies is very small). The summary effect A in this example is weighted more strongly towards A 1 than A 2 , since the weight of the study S 2 is very small: indeed, the adaptive layer improves the algorithmic performance by 0 . 64% for one instance, 5 . 71% for another instances, but by −0 . 33% for the last instance, and we are therefore less confident about the true effect of adaptiveness in this study (i.e., the standard deviation is large, and hence the weight is small, indicating that the estimate A 2 is less precise). The summary effect A reflecting the importance of the adaptive layer for the two small studies S 1 and S 2 is the weighted average of effects A 1 and A 2 :

Author contributions
Renata Turkeš carried out the literature review, identified and selected the studies, communicated with the authors of eligible studies via e-mail, collected and pre-processed the raw data into data_individiual_studies_filtered.zip , and wrote the manuscript. with its non-adaptive variant, collected in data_individiual_studies_raw.zip . Table 3 A toy example of a step-by-step meta-analysis with two included studies.