Update (1.2) to ANDURIL and ANDURYL: Performance improvements and a graphical user interface

This is an update to PII: S2352711018300608 and S2352711019302419 In this paper, we present three main improvements of ANDURIL and its python version ANDURYL. First the MATLAB version ANDURIL is brought to the Python version standard by implementing (i) user defined quantiles and (ii) the possibility to deal with missing values. Second, the computational engines of both ANDURIL and ANDURYL were significantly improved making calculation time lower and improving further accuracy. Finally a standalone Graphical User Interface is presented which we believe will make the software more accessible to practitioners of Cooke’s method

. Illustration of decision maker interpolation.  1. ANDURIL is brought to the Python version standard by implementing: (i) user defined quantiles and (ii) the possibility to deal with missing values. These features will not be discussed further. The reader is referred to [3] for an explanation of the main features now also available in AI v1. 2     The new code led also to improved accuracy of both AI and AY. That is, both solutions are closer to EXCALIBUR (CC). The differences between CC and AI and AY for the 7 studies where differences were observed, are shown in Table 2. This will be elaborated further below. 3. A standalone Graphical User Interface of ANDURYL is presented. A screen shot of the GUI is presented in Fig. 2

ANDURYL and ANDURIL code improvement
The main improvement in speed and accuracy is the result of a different implementation for calculating the Decision Maker's (DM) cumulative distribution function (CDF). In version 1.0 and 1.1, the DM's CDF was calculated by integrating the probability density function (PDF) of the weighted DM's numerically (quadrature method) through an anonymous function. Solving this integral is numerically expensive and when the probability density of one or more expert are very concentrated in a range in relation to that of other experts, parts of the PDF were skipped in the discretization used in the numerical integration.
In the new (AY v1.2 and AI v1.2), the old implementation of the integral is replaced by an interpolation of the CDF. As long as the PDF between the given quantiles is uniform (or loguniform), this gives the same results as solving the integral, but much quicker and without inaccuracies due to the discretization of the integral. Fig. 1 illustrates the process of interpolation for the decision maker.
Note that the DM quantiles (''DM full'' in the figure) are determined by interpolating each of the (two in this case) experts' answers (following the dashed lines). This results in the full detailed CDF of the decision maker. This can subsequently be interpolated at the percentiles of interest (which is EXCALIBUR's output). Note that the interpolation is not carried out over the quantile direction.

ANDURYL GUI
The main improvement for the Python version is the graphical user interface. This interface, programmed with the Python module PyQt5, is compiled with PyInstaller (for Windows), such that it is a stand-alone executable. This makes ANDURYL accessible to non-Python users. The layout of the user interface consists of 4 overviews, for the experts, items, assessments and results, as shown in Fig. 2.
The following list gives an overview of the functionalities that the stand-alone GUI offers: • Assessments per expert or item can be plotted as a PDF, CDF, survival function or range. The CDF option is shown in Fig. 2 on the foreground.
• Because of the improvements in computational performance, it is now less demanding to do a robustness analysis for excluding multiple experts or items. The results of the robustness analysis can be shown in box plots.
• The program has options for saving the project in EXCAL-IBUR format or a more common JSON format.
• Separate DM's results, such as the full CDFs, can be exported or copied to clipboard.
• The AY code is separated between calculation and user interface functionalities so that the Python-module can also be used from a script or Jupyter notebook. For research purposes this is a useful functionality.
• The fact that AY is still significantly faster than AI, as shown in Table 1, is due to differences in implementation. In AI several expensive operations are re-calculated for different iterations. In AY the amount of data that is re-calculated is minimized.

Comparing with previous studies
In [4], 33 post-2006 studies using Cooke's classical method are presented using CC. We use these data to compare output from AY and AI to both CC, the MATLAB implementation AI of the v1.0 paper [2] and the Python implementation of the paper [3].
The differences are smaller compared to the results from the last code version. For two studies, ''Hemophilia'' and ''Ice sheets'' the differences are still significant. For four other studies the results seem to be due to rounding errors. Of the remaining 26 studies, the majority have equal results. Table 2 shows the differences for the studies where differences are still observed.

Conclusions
The Python module named ANDURYL (AY) has been extended with a graphical user interface and is available as stand-alone executable. The MATLAB toolbox named ANDURIL (AI) for combining expert judgments applying Cooke's method has been further extended by adding functionalities for user defined quantiles and handling missing values. The stand-alone GUI enables practitioners and researchers that have no Python or MATLAB experience to apply Cooke's method with ANDURYL. For users that are more familiar with programming, the MATLAB toolbox and Python GUI are a means to perform or analyze expert elicitations in a reproducible way. The improved speed and accuracy contribute to this cause. Both codes are open source to encourage usage and further development.

Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.