Development, test and comparison of two Multiple Criteria Decision Analysis (MCDA) models: A case of healthcare infrastructure location

,


Introduction
The National Health Service (NHS) in the UK has a requirement to develop reliable, robust and transparent operational decision-making processes; and, when appropriate, to include the local population within the process (Department of Health (DoH), 2010).Within the planning of new healthcare infrastructure such as hospitals, clinics or healthcare centres, a key operational decision is the choice of the location.The infrastructure site decision influences and shapes the overall healthcare network within an area.It is also a delicate decision for the local population who are traditionally consulted via a mailed questionnaire.However, this method is not the most rational or transparent way for optimising the location and other methods are needed to improve the site location decision-making process (Dehe, Bamford, Bamford, & Moxham, 2011;Feldmann & Olhager, 2013).Formal decision-making models and intelligent systems can be used to support the decision-making processes and it is suggested, in this paper, that MCDA models are appropriate techniques to resolve the location issue, especially when considering the NHS environment and the objectives set by the stakeholders: robustness and transparency.Healthcare organisations are becoming increasingly accountable to the local population (DoH, 2010) and modelling techniques such as location-allocation models and Geographic Information Systems (GIS) have been promoted to optimise site locations (Rosero-Bixby, 2004).However, it is suggested that these types of modelling technique do not entirely satisfy the transparency and inclusivity objectives of the NHS; it is difficult to simulate and model the more qualitative criteria and inputs gathered from the 'Voice of the Customer' (Bamford & Forrester, 2010).
This paper reports on the empirical differences perceived between the operational application of ER and AHP, when applied to the healthcare site selection, as well as identifying some of the managerial and practical implications for the decision-makers.The research make a practical contribution rather than a purely theoretical one, hence the technical and mathematical background will not be considered, and rather it is the socio-technical aspect of the models implementation that are the focus in this paper (Singh & Wood-Harper, 2011).To direct the study two specific research questions (RQ) were developed: RQ1: Are the operational processes and outcomes significantly different according to the MCDA model implemented: ER or AHP; RQ2: According to the decision-makers, what is the most reliable and appropriate modelling techniques to provide a rational, inclusive and transparent operational solution?
2. Literature -MCDA in context and practise

Modelling theory and roles of models
Various model definitions have been discussed over the years, for instance Ackoff and Sasieni (1968) defined a model as a representation of the reality.However, Pidd (2003) explained this simplistic definition did not address the fact that people have different worldview and perception of the reality, as well as that a model can never be entirely complete and accurate.Therefore, Pidd (2003) preferred defining a model as an external and explicit representation part of a reality as seen by the decision-makers and modellers.This means, models are an approximation of the reality and that according to the specific model used to look at a real world problem the processes and outcomes might be different.In this paper, it is intended to establish, whether by looking at the same real world problem -the site location for a new healthcare centre -throughout two different MCDA models: ER and AHP, the processes and the outcomes are different or not, and whether one is more appropriate than the other, in this particular setting.
According to Box and Draper (1987, p.424) ''Essentially, all models are wrong, but some are more useful than others''.Hence, models have different characteristics, and one may want to identify the most appropriate model to use for solving a specific problem in an identified environment.To identify the most appropriate model, one may want to look at: (i) the robustness and the representativeness of the results generated, which are measures and perceptions of accuracy; and (ii) the repeatability and the reproducibility, associated with the consistency and transparency, which are measures of precision of the model and its process (Breyfogle, 2003).For instance, will the models allow the decision-makers and participants to be consistent at a different time?And, how representative of the perceived reality are the results?Moreover, the consistency, transparency and the facilitation or practicality must be taken into account when implementing a model.Fig. 1 illustrates an assessment framework to determine what model would lead towards the optimum solution.MCDA Ram, Montibeller, and Morton (2011) and Golmohammadi and Mellat-Parast (2012) stated that when strategic options are being evaluated for instance in supplier or location selections, MCDA is the suitable approach to handle conflicting and both qualitative and quantitative objectives.MCDA provides a framework to aid with making complex decision by creating a platform where all stakeholders can share information, in order to develop a consensus or find a compromise.The sequence of tasks becomes logical, first by structuring the problem; second, by modelling the criteria preference and their importance; then, by aggregating the alternatives evaluation; and finally allowing the decision to be made (Saaty, 1980;Santos, Belton, & Howick, 2002;Yang, 2001).Ren, Gao, and Bian (2013, p. 3) pointed out that, from a mathematical perspective, a MCDA model is defined by a set of alternatives, denoted by A = {a1, a2, . .., am}, from which a decision-maker will select the optimal alternative, according to the identified set of criteria, denoted by C = {C1, C2, . .., Cn}.Also, an interval weight vector, denoted by X = (x1, x1, . .., xn), will be given, where xj = [xL j, xR j] (j e N = {1, 2, . .., n}) and 0 6 xL j 6 xR j 6 1.This represents the relative importance of each criterion.Tavana and Sodenkamp (2010) explained that MCDA enables the stakeholders to create a framework to exchange their information and knowledge while exploring their value systems through the weighting and scoring mechanisms.Furthermore, Ormerod (2010) suggested that different frameworks and mechanisms inform the stakeholders' beliefs about the relationship between the options and the outcomes.While, Belton and Stewart (2002) explained the myths of MCDA, emphasising that there are no right answers due to the subjectivity of the inputs.The subjectivity is inherent to the choice of criteria, the weighting and the assessment.Therefore, according to the framework selected, the subjectivity might be different, even when the common final aim leans towards a transparent, informed and sensitive decision.Xu and Yang (2001) wrote that there are many methods available for solving MCDA problems.Amongst the most theoretical and empirically sound techniques, there are ER and AHP (Guo, Yang, Chin, & Wang, 2007;Saaty, 1980;Saaty & Vargas, 2001;Xu & Yang, 2001).Other methods which can be found are: TOPSIS, VIKOR, ELECTRE, and UTASTAR (De Moraes, Garcia, Ensslin, Da Conceição, & De Carvalho, 2010;Grigoroudis, Orfanoudaki, & Zopounidis, 2012;Liao & Xu, 2013;Santos et al., 2002;Yang, 2001).The literature reports several applications of MCDA.Some applications are associated with a sector of activity; manufacturing, healthcare or construction.Other applications are related to a specific type of decision.Is one technique more appropriate than another, in a specific context?

MCDA in healthcare
The literature shows a worldwide use of MDCA in the healthcare sector.Its use and applications remain varied, to support both clinical (Miot, Wagner, Khoury, Rindress, & Goetghebeur, 2012;Tony et al., 2011;Youngkong, Teerawattananon, Tantivess, & Baltussen, 2012) and managerial (De Moraes et al., 2010;Dey, Hariharan, & Clegg, 2006;Grigoroudis et al., 2012;Kornfeld & Kara, 2011) decision-making during complex problem solving.Büyüközkan, Çifçi, and Güleryüz (2011) showed how a fuzzy AHP model supported the evaluation and the perception of the service quality in a Turkish hospital; they determined the factors and criteria that hospitals should focus on to optimise their service quality.

MCDA in site selection
Site selection is a critically strategic decision as it could potentially make or break a business, independently of the industry because location decisions involve long term resource commitment and have significant impacts on the operations strategy and the key operations performance indicators such as cost, flexibility, speed and dependability (Ertug ˘rul & Karakas ßog ˘lu, 2008; Salles, 2007;Yang & Lee, 1997).The literature is very diverse regarding site selection or facility location, however, as for complex processes it requires rationalised decision-making, often subject to uncertainty (Hodgett, Martin, Montague, & Talford, 2013).There are numerous MCDA applications in the site selection problem; this is one of the first problems studied in the MCDA literature for instance with the work of Keeney and Raiffa (1993) where they explore airport location.Furthermore, several papers have been published regarding landfill site selection considering the economic, ecological and environmental issues associated with the decision, often the MCDA models were associated with the use of Geographic Information System (Fatta, Saravanos, & Loizidou, 1998;Gorsevski, Donevska, Mitrovski, & Frizado, 2012;Guiqin, Li, Guoxue, & Lijun, 2009;Onut & Soner, 2007).Other papers, less specific, presented MCDA models for other infrastructure location, for instance, Chen (2006) explained the complexity in the convention site selection and suggested AHP as a method to support the decision by making sense of the multitude of variables encompassed; they demonstrated the use of their five criteria and 17 sub-criteria model within a site selection in Taiwan.Ertug ˘rul and Karakas ßog ˘lu (2008) chose to demonstrate the MCDA application to optimise the facility location of a textile organisation in Turkey.
However, case studies investigating the healthcare site selection problem, using MCDA, are limited.It was identified that, in their paper, only Vahidnia, Alesheikh, and Alimohammadi (2009) developed an AHP model to find the best site for a new hospital.Their model has five criteria: distance from arterial routes, travel time, contamination, land cost and population density.
Additionally, very few studies comparing results between different models were found.Only Ertug ˘rul and Karakas ßog ˘lu (2008) compared the AHP method with TOPSIS, and Zhang, Wang, Sun, and Wang (2011) who compare their methods with two different authors Beynon (2002) and Hua, Gong, and Xu (2008) methods, which lead them to observe contradictory results.This is noted, despite the common recognition of the compensation involved in any aggregation models and the subjectivity incurred in a framework.For example, Grigoroudis et al. (2012) explained that results are affected by both the model reference sets and by the decision-makers consistency and interpretation of the model mechanisms.In their paper, however, Ertug ˘rul and Karakas ßog ˘lu (2008) contrasted two modelling techniques: AHP and TOPSIS, and concluded that, despite that both AHP and TOPSIS having their own characteristics, the ranking of the three alternatives was the same.They demonstrated that, when the decision-makers were consistent, both methods could be appropriate, even if they recognised that decision-makers should choose the methods fitting the problems and the situation.However, the study did not address the process differences and preferences of the decision-makers in great depth, and this is the reason why it will be attempted here in comparing two methods: AHP and ER, and evaluate the managerial consequences of choosing one or the other.
There are many methods available for solving MCDA problems, however, some methods were criticised for lacking theoretical soundness and empirical evidence (Xu & Yang, 2003).Nevertheless, both ER and AHP are both theoretical and empirical grounded (Saaty, 1980;Saaty & Vargas, 2001;Xu & Yang, 2003).Therefore, it was useful to test whether or not by implementing this two different MCDA models the optimisation of the decision-making process of the site selection was going to be affected.

Evidential Reasoning ER and its application
The ER approach is amongst the latest MCDA technique, developed to handle uncertainty and randomness.Xu (2011), Liu, Bian, Lin, Dong, andXu (2011) and Wang and Elhag (2008) stated that the ER was first developed by Yang and Singh (1994) to solve multiple criteria decision problems taking into account qualitative and quantitative attributes as well as the inherent uncertainty, by combining the Dempster-Shafer (D-S) theory (Shafer, 1976) with a distributed modelling framework.The difference with the other more traditional MCDA models is that ER uses an extended decision matrix in which each attribute of an alternative is described by a distributed assessment using a belief structure (Liu et al., 2011;Xu & Yang, 2001;Xu & Yang, 2003).For instance the distributed assessment results of the sub-criteria regeneration impact for alternative A can be {(Best, 33%), (Good, 33%), (Average, 33%), (Poor, 0%), (Worst, 0%)}, whereas for B it can be {(Best, 0%), (Good, 50%), (Average, 50%), (Poor, 0%), (Worst, 0%)}.ER uses a Simple Additive Weighting as scoring methods to calculate the overall score of an alternative as the weighted sum of the attribute scores or utilities (Xu, 2011;Xu & Yang, 2001;Xu & Yang, 2003;Yang, 2001).This process can be facilitated by the Intelligent Decision Systems (IDS) software developed and tested by Yang and his collaborators between 1998and 2006(Wang & Elhag, 2008;Xu, 2011;Yang 2007).Xu and Yang (2001), Xu and Yang (2003) also clearly explained that by using a distributed assessment technique decision-makers can capture the diverse type of uncertainties and model subjective judgement.Hence, they clarified that ER approach uses the Dempster-Shafer (D-S) theory as aggregation mechanisms; Bi, Guan, and Bell (2008) explained that the D-S theory is an appropriate and suitable approach for dealing with uncertainty and imprecision.It provides a coherent framework to cope with the lack of evidence and discard the insufficient reasoning principle.ER enables to translate the relationship between the object and the degree of goodness or badness of its sub-criteria, which is measured by both ''the degree to which that sub-criteria is important to the object and the degree to which the sub-criteria belongs to the good (or bad) category'' (Xu & Yang, 2001, p. 8).Furthermore, it allows decision-makers preferences to be aggregated in a structured and rigorously without accepting the linearity assumption (Chin, Wang, Yang, & Poon, 2009).This makes to some extent ER different from other MCDA approach such as AHP or TOPSIS (Ertug ˘rul & Karakas ßog ˘lu, 2008; Seçme, Bayrakdaroglu, & Kahraman, 2009;Zhang et al., 2011).Furthermore, ER has been applied in different sectors and industries construction, security, transport, and IT, with diverse applications such as supplier selection, performance measurement, assessment, risk management, new product development, and data aggregation (Chin, Xu, Yang, & Lam, 2008;Liu et al., 2011; Wang &  Elhag, 2008; Wang, Yang, & Xu, 2006; Yang, Wang, Bonsall,  2009; Zhang, Deng, Wei, & Deng, 2012).However, not many publications were found in the Healthcare sector, only Tang et al. (2012) used ER in order to assess and analysed the risks in an NHS organisation.

Analytical Hierarchy Process AHP and its application
AHP is a general theory of measurement; it is an effective approach to handling decision-making and certainly the most popular MCDA methodology (Bozbura, Beskese, & Kahraman, 2007;Chen & Huang, 2007;Jakhar & Barua, 2013;Kang & Lee, 2007;Partovi, 2007).It was developed by Saaty in the 1980's for resolving unstructured problems in any disciples or business areas (Wu, Lee, Tah, & Aouad, 2007).Saaty and Vargas (2001) explained that it was designed to cope with the uncertainty, and to optimise the evaluation the available alternatives.By undertaking pair-wise comparison judgments and aggregating the scores, a ranking of alternative is developed.The advantage resides in the fact that it allows inconsistency to be assessed but simultaneously improving the consistency of the decision (Saaty & Vargas, 2001).
The logic behind AHP is in building a three level hierarchy model with the goal, the criteria and the alternatives to be assessed.Cousins, Lamming, Lawson, and Squire (2008) explained that to express the relative importance of one criterion over another AHP uses the pair-wise comparison method.The scale can be selected to accommodate the needs of the decision-makers as Tiwari and Banerjee (2010) demonstrated.We have used the fundamental five levels scale to offer a wide range of possibilities as Table 1 shows.This fundamental scale was defined by Saaty and Vargas (2001), and has been theoretically justified and its effectiveness validated.This scale is used with the reciprocals values when the relationship between two activities is inverted.Belton and Gear (1983), Chin et al. (2008) and Taround andYang (2013, p. 1222) recognised the excellence of the AHP approach.However, they also explained that it has a number of limitations.Firstly, as AHP treats criteria weights and scores in the same way, applying pair-wise comparison, which, they believed, leads to ranking reversal problems, moreover, one needs to be concerned with the number of judgments required to derive relative priorities, which can create inconsistency issues (Mustafa & Al-Bahar, 1991).Furthermore, AHP lacks the capacity to cope with uncertainty.Finally, the introduction of new criteria, or alternatives, will require the modification of the whole model (Belton & Gear, 1983;Belton & Stewart, 2002).The limitations of AHP do not undermine its usefulness, but have stimulated researchers to develop alternative techniques, such as ER (Taround & Yang, 2013, p. 1222).
To solve the developed model, a software called 'Make it Rational' (MiR) was used (http://makeitrational.com/).This allowed comparing 'like-for-life' modelling techniques; it was felt that by not implementing both models via a software interface the results could have been compromised or at least biased toward one or another model.

General differences and similarities
Evidential Reasoning (ER) and Analytical Hierarchy Process (AHP) were the two approaches presented and selected, because it was considered that AHP was the most popular approach, and ER was an excellent complementary approach.However, the researcher recognises and acknowledges the other powerful techniques, such as VIKOR and TOPSIS.ER and AHP major practical differences reside in the assessment level and in the assessment technique.ER focuses on the sub-criteria level of the model, uses a degree of belief for the assessment, and the Likert scale for the weighting; whereas AHP focuses on the aggregate criteria and uses pair-wise comparison, as Fig. 2 illustrates.These differences influence the subjectivity within the modelling process, and may lead to practical and managerial implications.
Both ER and AHP use equivalent hierarchical structures therefore one can follow the same process with the identified group of stakeholders to satisfy the accountability objectives by engaging with the stakeholders.However, the differences will take place in the weighting and scoring phases.The assessment of alternatives follows different type of mechanisms.Also, one can wonder whether, by using one or the other method, it will influence the results interpretation.

Methodology
This research used an embedded single-case study in order to develop, test and compare the two MCDA models (Yin, 2009).The research is designed around a series of eight workshops; it was adopted to gather rich data in order to develop an understanding of the use of Evidential Reasoning and Analytical Hierarchy Process, as well as understanding the socio-technical processes informing the final location decision.The researchers had a direct access to the organisation over an extended time period: two years, and this experiment lasted about 6 months where eight workshops were organised and attended by the different groups of stakeholders.There are shortfalls associated with a single case study; these are often related to the external validity and the generalisation (Gay & Bamford, 2007).Nevertheless, it remains a popular research methods and many important operational concepts have been developed by using a case study approach (Voss, Tsikriktsis, & Frohlich 2002).The case study is therefore a valid method to contribute to the body of knowledge by developing an understanding of the causal mechanisms of a particular phenomenon (Yin, 2009).

The case study
The model structure and hierarchy was developed in collaboration with an NHS organisation in the UK.This healthcare organisation commissions the full range of clinical services throughout 58 community-based health services across 100 sites, within the 30 Local Authority wards, each of them with a population of about 17,000 inhabitants.The total catchment area represents approximately 500,000 people, living in both urban and rural area.The organisation has set particular priorities: for instance, the reduction of health inequalities, the improvement of the clinical quality and safety, as well as increasing the patient experience through enhance efficiency and effectiveness performances.This can be accommodated by a move toward more community-based care provision, as specified within Lord Darzi's Report (2007).The evidence favouring one activity over another is of the highest possible order of affirmation However, to achieve these objectives the organisation has to undertake extensive infrastructure development over the next decade, but lacks mechanisms, systems or procedures for overseeing their planning and ensuring that the organisation's future strategic needs are achieved.

The research rationale and process
The objective was to optimise healthcare site selection decision-making processes within a National Health Service (NHS) organisation in the North of England.It was also aimed at establishing the most reliable and appropriate modelling techniques to tend toward a rational, inclusive and transparent solution, which fits with the key objectives and indicators of the organisation.For these reasons the two research questions mentioned, in the introduction, were developed.In the first instance, the MCDA model was developed with a wide range of carefully selected stakeholders, and was subsequently validated.The assessment was conducted by the team of experts to reduce the information asymmetry and be as informed and sensitive as possible.The AHP assessment was undertaken concomitantly using Make it Rational (MiR) software in order to answer the stated research questions and be able to compare ER and AHP as objectively as possible.This process was undertaken as an experiment.

Data collection to build the models
A substantial data set from public consultations was considered with both qualitative and quantitative information that supported the design of the final MCDA model as Fig. 3 shows.Furthermore, data were gathered from four specific workshops compile the final model and solve it using ER and two extra workshops were set up for solving the AHP model.These workshops were organised to capture both the 'voice of the local community' and the 'experts judgment' such as: Estates, Primary care, Planners, Clinicians and other key decision-makers from the senior management.These sets of data were used to identify and agree the seven criteria: environment and safety, size, total cost, accessibility, design, risks, and population profile; and the 28 sub-criteria and their associated weightings (c.f.Fig. 3).Therefore, in total six facilitated workshops, which involved a total of 45 stakeholders, enable the authors to collect qualitative and quantitative data to be able to compile and solve both final ER and AHP models as the findings section will present, an extra two workshop were held to compare the models.

Reflection on the approach using semi-structured interviews and group discussions
In order to answer the second research question, and identify the most reliable and appropriate modelling techniques, it was relevant to gather information directly from the decision-makers, who, in the future, will own the process.The authors were keen to collect qualitative information regarding both processes ER and AHP during and after the experiments.The rational was to understand what are the models' characteristics that the decision-makers require to optimise the process.Therefore, semi-structured interviews and group discussions were organised, during the last two workshops, around the following questions: 'what did think of ER?'; 'How was the APH process?'; 'How did you find the pair-wise comparison?'; Between ER and AHP, which one did you prefer?';Building the ER model, was it cumbersome?'Did you feel that your opinion was well integrated within the final AHP model?';Overall was ER and AHP a complex process to go through?'.Moreover, during both processes ER and AHP, the authors made observations regarding the interactions and the dynamic between the decision-makers.It was important to perceive how the stakeholders and decision-makers responded during the processes.This information was recorded to support the discussion in this paper.
This paper reports an experiment of applying two different MCDA techniques: ER and AHP, to optimise the healthcare site selection.It had for objectives to establish (i) whether there is a difference between the two models processes and results and (ii) identify what would be the optimum process within this environment.These objectives were achieved by combining both quantitative and qualitative data in order to develop and solved the models and qualitative data in order to gather the perception of the decision-makers and establish the most reliable and appropriate modelling techniques.

The ER model
From the facilitated workshops, which involved a total of 45 stakeholders, it was possible to compile the final model, with the associated weightings, which were the rounded average of each individual score, as shown in Fig. 3 below.This model is composed of seven criteria and 28 sub-criteria.As mentioned previously, in the ER approach the assessment takes place at the sub-criteria level, therefore, it was required to identify whether the sub-criteria are evaluated quantitatively, noted Q, or qualitatively, noted QL in Fig. 3. Once the weightings were identified and validated they were normalised, which are used in this analysis further.The normalisation process helped to compare the results generated by the two models.

The AHP model
The AHP model has traditionally three levels: the goal, the criteria and the alternatives as illustrated in Fig. 4. The set of the seven criteria is the common structure, as it is independent of the selected modelling techniques.2. This was established by the group of decision-makers, and is consistent with the weighting determined in the ER model.

Location of New healthcare centre
While ER uses the Likert scale from 1 to 10 (10 being the highest) to identify the weighting of each criteria, AHP uses pair-wise comparison, modelling (7 Â 7) 49 relationships.This means for instance that: Environment and safety is equally preferred to Population profile; then a value of 1 is captured; Size is strongly non-preferred to cost; then a value of 1/5 is captured.This is also automatically recorded in the reciprocal cell as Total cost is strongly preferred to Size where a value 5 is registered.Once this is averaged and normalised, the weightings are obtained.
It is important to note that AHP model logic works at the highest level (e.g.: at the criteria level).The sub-criteria are also taken into account by forming the overall definition of each criterion.It would have been impractical to facilitate the pair-wise comparison process to identify the weights and the assessments for the 28 sub-criteria and the decision-makers would not have bought into the process; a criticism of AHP.

ER and AHP the weightings
In Table 3, the weighting assigned for both models are presented.Note that the weighting range is different whether ER or AHP was the selected framework to solve the problem.With ER the weighting are included into a range from 8.90% to 22.22%; whereas with AHP the range is wider from 3.53% to 38.89%.From the assessment it was established that when stakeholders use a Likert scale from 1 to 10 it is likely that little difference can be perceived between the criteria but that the uniformity is respected and it is highly transparent.However, using pair-wise comparison, the difference is amplified, but there is room for inconsistency when criteria are being compared against other criteria, and stakeholders might have a less transparent perception of the weighting phase.The second relevant point to mention is that in this case using ER or AHP led to the same ranking, which is positive, and translate that the decision-makers were consistent in their approach, and gave confidence to proceed with the comparison.

ER and AHP the assessments
The next step was the assessment of alternatives: A and B in this case; which allowed the ranking of the alternatives.With ER the degree of belief for each sub-criteria is established independently, whereas AHP remains at the criteria level and assessed the alternative against each other using the pair-wise comparison.Table 4 compiled the results from both assessments at the criteria level; note that even if the results provided shows that location A is significantly the preferred option in 3 criteria, and location B in 2 criteria, and that overall A is the preferred option, the quantification differences which is the most paramount indicator for the final decision is substantially different according the selected modelling approach.Therefore, for this reason a statistical test: 2 Proportion Test was undertaken.Hypothesis testing: is there any significant difference between the results scoring range of ER and AHP? H1: proportion [ER(a À b) -AHP(a À b)].P value < 0.05 (with a = 0.05), hence, we can be 95% confident that there is a difference between the results from ER and AHP.
With ER, it is suggested that both alternatives reach similar scores (A = 56 and B = 54 or normalised A = 51 and B = 49), it can be interpreted as location A and B are performing similarly; however, using the AHP model, there is less doubt that alternative A significantly outstrips the alternative B (A = 62.35 and B = 37.65 normalised).Having said that, this does not indicate which model provides the optimum solution in this example and in this context.
Would the final recommendations change based on the ER or AHP results?What are the most suitable models to optimise the decision-making process in this environment?

Discussion
The framework developed from the literature (Fig. 1) structured the assessment of both models and focussed the discussion of their implications, from a practical and managerial perspectives.The following section will deliberate the models and compare them against the framework criteria: the process consistency, the process transparency and its facilitation and access for the decision makers; as well as the robustness of the result and their representativeness.The rationality element was not considered as a criterion on its own, but rather as an aspect linked to each of the five criteria transversal to both the process and results.This was achieved by the reflective work that the authors have undertaken with the decision-makers during the last two workshops.

Processes and their precisions
The processes for weighting and assessing the criteria had to be consistent, repeatable and transparent, because they were used by the group of decision-makers at different times.To test this, parts of the processes were selected and tested by asking the stakeholders to re-weight and re-assess criteria and alternatives, in order to establish to what extent the same weightings and assessments could be reproduced, and to test the capabilities of the measurement models (Breyfogle, 2003).This goes some way towards addressing the concerns regarding the myths of MCDA, which state that it does not always provide a consistent answer, as Belton and Stewart (2002) suggested.Moreover, different groups of stakeholders were asked to weight and assess the same criteria and alternatives, based on the same given information, to establish whether the differences were significant or not.This relates to view MCDA as being highly useful for exchanging knowledge (Tavana & Sodenkamp, 2010).
By using ER, the weighting and assessment processes generated good consistency.Over time, participants were able to repeat their assessments, quite confidently, by using the Likert scale methodology.However, by using AHP and the pair-wise comparison, the process was found to be less consistent, especially as the model became bigger anomalies and contradictions were created, as observed the decision-makers.This could partly be explained by the decision-makers not being familiar with pair-wise comparison methods, considered confusing by the group of participants.Therefore, it can be suggested that ER is more likely to be a consistent method for assessing alternatives, but could lead to some inconsistency within the weighting process, as participants and decision-makers were reluctant to use the whole scale and the range of most of the weightings were only between 6 and 9 on the entire, 1-10 scale, which could affect the final results.
Therefore, in terms of consistency, it was recommended that the pair-wise comparison is used at the criteria level, and the degree of belief technique is used in the assessment, so as to reach an optimum process consistency.
Transparency was the primary criteria for justifying the MCDA route, as discussed earlier in this paper.The objectives were to embed inclusive processes and make them easy to understand for the large range of stakeholders involved.In this case, ER seemed easier for the majority of the participants involved; ''ER was more straightforward than AHP'' according to the participants.This is reinforcing the findings from the literature, which states that ER is a 'simple' process, and that there are many different ways to compile and aggregate the results, as Xu and Yang (2001), and Xu (2011) explained.Also, the pair-wise comparison had to be established by a consensus, and some of the stakeholders and decision-makers found it slightly confusing and rather redundant, which reduced the transparency factor.Hence, it was confirmed that, for the large range of stakeholders, ER was more a transparent process than the pair-wise comparison.It was easier to track, as the individual inputs could be highlighted, as part of the process is to average the different scores given by all the participants, and the process allows the average scores to be reproduced on different occasions.By using AHP, it was necessary to identify the pair-wise weight, or assessment, based on the general consensus given at the specific time.However, it was found that it did not keep track of what happened during the process, which could, arguably, make it less transparent than ER.Possibly, to overcome this issue, when using AHP, every stakeholder could provide their own pair-wise comparison and an aggregated mean of the individual judgment could be generated.However, this was considered impractical at the time of the experiment but will be extremely relevant in the future when the MCDA maturity of the organisation has grown.
Both models can be facilitated using a large number of stakeholders.It was felt that AHP was easier and faster, as it interacts with a higher level of the structure.Moreover, AHP uses one mechanism for both weighting and assessing (i.e.: pair-wise comparison); whereas, ER uses the Likert scale for the weighting, then the degree of belief for the assessment of the alternatives.More time needed to be allowed for facilitating ER as opposed to AHP.AHP was also easily facilitated by an excel spread sheet, which proved convenient for the decision-makers.Having said that, from the feedback received, participants were more comfortable using the Likert scale and degree of belief system than pair-wise comparison, despite the training provided beforehand.It was felt that AHP was more accessible, as it remained at the aggregate level of the hierarchy model -very useful for unstructured problem solving -whereas, ER goes down to the smallest level of the model; in this case, the sub-criteria (Saaty, 1980;Wang et al., 2006).

Results and their accuracy
The robustness of the results was hampered by the possibility of introducing bias; the stability of the models and the sensibility aspect of the results were other factors considered.Ideally, the model needed to be bias proof and sensible enough to adequately translate the results.It was suggested that, potentially, AHP was the more sensible option, as the spread of the results shown; however, it was more likely to introduce bias into the results, by finding consensus based on the strongest personality in the room, while weighting and assessing criteria as well as the alternatives.
Moreover, the AHP method could possibly introduce unsteady elements by not following a logical and consistent pair-wise assessment, and there is a danger that contradictions might be input into the model.Both techniques provide sensitivity analysis.This translates the robustness of the results, as one can further understand what the ranking means, plus what influence changing a weighting, or unit of assessment would have on the results.Therefore, , it was analysed that ER was less subject to bias and was slightly more stable than AHP, perhaps because it works at the lower level of the model, in line with the observations of Xu and Yang (2001).
It was also important to evaluate if the model distorts reality by appreciating the level of subjectivity.The mechanism, for establishing whether or not this was the case, was to compare the results of the model against other measurements.In this case, the measurement available was the extensive survey of N = 3055, undertaken by the organisation during the public consultation, from which 92% of the participants were in favour of Location A. The AHP model shows a wider range between A and B with 24.7 points of difference, whereas ER model only established a 2 points difference in (both cases normalised (56-54) and non-normalised (51-49) as illustrated in Table 4.
The rationality aspect was defined by asking the stakeholders what was the process they perceived the most rational, ER came out in front: ''ER seems more scientific'' argued one the decision-makers.However, from the example AHP results seem to be more in-line with reality; in the past the questionnaire was the tool used to make the final decision for the site locations.In this case the reality was translated more through the AHP model than with the ER model.As mentioned previously, this was due to the tendency that with ER, only a part of the scale was used (6-9), especially during the weighting stage, this was clear in the presented case, which is translated by the small range for the criteria weighting varying from 8.90 to 22.22 (once normalised) whereas with AHP, it fluctuates from 3.53 to 38.89 (see Tables 2  and 3).This has had a substantial impact on the results.Hence, it was felt that the AHP model was appropriate to translate better the reality, as seen by the local population, thanks to its criteria pair-wise comparison element.

Research questions answered
To provide specifically focused conclusions and evidence-based the originality of the paper, the research questions are answered each in turn.RQ1: Are the operational processes and outcomes significantly different according to the MCDA model implemented: ER or AHP?There were significant differences between the process and the outcomes of the two models.According to the model selected, the results were statistically and significantly different; thus, this would have impacted the final decision.The process selected also impacts the practical and managerial implications and behaviours for both the participants and decision-makers.ER uses different methods for weighting and assessing and works at the lower level of the model, which supports the transparency and robustness elements; whereas, the decision-makers found AHP to be more flexible, very efficient and extremely relevant in a smaller strategic committee, in which the level of transparency for the local population was not necessarily the prime issue.Moreover, the pair-wise comparison seemed more appropriate  for grasping the real, or subjective, differences.This section reinforces the quote, stated at the beginning, that ''essentially, all models are wrong, but some are more useful than others'' (Box & Draper, 1987).From the findings, it was felt that this is greatly dependent upon the environment.
With regards to the second question, RQ2: According to the decision-makers, what is the most reliable and appropriate modelling techniques to provide a rational, inclusive and transparent operational solution?The most reliable and appropriate modelling technique to use in the specific context of site selection for future healthcare infrastructure, when seeking a rational, inclusive and transparent solution, would be a hybrid version of both ER and AHP.It was agreed that both models were reliable techniques with different characteristics.Thus, to optimise both the process transparency and consistency, the use of ER, merged with the AHP pair-wise comparison at the criteria weighting process, is suggested.It is believed that, by implementing this hybrid version, the rationality of the decision can be optimised even further, by developing an informed, sensitive and transparent decision for the site selection.Consequently, it is recommended to work at the lower level of the model, as ER suggests, in order to reduce the information asymmetry; but that the weightings of the criteria are established, through the pair-wise comparison, as AHP implies, and as is illustrated in Fig. 5.This goes towards Zhang et al. (2012) who also used a mixed AHP and ER approach to propose a flexible and practical model to cope with qualitative and quantitative data as well as with uncertainty for the assessment of e-commerce security.
Therefore, by solving this hybrid model, using pair-wise comparison to assess the criteria weights and the degree of belief to assess the alternatives, the normalised results are that A is the preferred options with 56%, and B has a total score of 44%.

Practical contribution, limitations and further research
The use of these models directly influenced the board of directors of this National Health Service (NHS) organisation to make an informed operational decision for the location of the £15 million health centre.As several attributes were conflicting these techniques were useful to aggregate the different stakeholders' perspectives and to reach agreement in selecting the key factors in identifying the optimum healthcare centre location.By going through this process the healthcare organisation became more informed and sensitive in appreciating the alternatives' differences; ultimately this allowed a more rational ranking of alternative by preferences.It has also been beneficial to the future patients, who were able to follow and take part in the evidence based decision-making process.This paper makes a defined technical and practical contribution by examining the use of MCDA models in operational location decision-making, and by evidencing the most relevant model via a thorough comparison.Furthermore, the model structure is being used as a starting point to replicate future infrastructure selection decisions, which has been a long standing issue.To put this into perspective, over the next six years, ten new schemes are planned in this specific organisation, representing more than £150 million of investment.For this reason, the site selection and location decisions will be scrutinised and the emerging hybrid methodologies will help provided effective and efficient guidance.
The authors appreciate that ER and AHP have different inherent characteristics and assumptions, hence the comparison at a theoretical level could be difficult to justify; however, the comparison is meaningful at the practical and practitioner levels, as decision-makers use the model to support the complex operational decisions to be resolved.Therefore, according to the decision-makers one method can be better than the other.This research has evidence-based that the proposed hybrid version leads to more optimum operational solutions and a more seamless process from the decision-makers perspectives, than the traditional ER or AHP.The advantage is that decision-makers can gain enhanced confidence in the results generated by the model and can justify further the reasons for the model characteristics.
There are a number of potential areas of further operational research in order to enrich this study and overcome its limitations.Firstly, it is relevant to facilitate the pair-wise comparison individually and develop a geometric mean of the assessments, within AHP, instead of seeking a general consensus.However, as explained, this would have been impractical at the time of the experiment; nevertheless, it will be considered in the future.Secondly, the perceptions of the decision-makers were gathered qualitatively, it might be appropriate to develop and validate a construct, in order to measure quantitatively aspect of rationality and transparency for each model.Thirdly, it would be relevant to test this new hybrid model from the beginning of the process, and compare the results with the ER and AHP models.These suggestions would strengthen the validity of the results presented in this paper.Finally, to explore further the phenomenon of this hybrid model and strengthen the impact to Expert and Intelligent Systems, this technique will be tested, as part of future research, in different sector and for different type decision-making.These findings should also encourage Expert and Intelligent Systems researchers to compare other MCDA techniques such as TOPSIS, VIKOR, ELECTRE, and UTASTAR, in order to establish optimum combination characteristics.
In the AHP model, the weightings of criteria are pair-wise compared and the results are shown in Table

Table 2
AHP Pair-wise comparison table for the criteria weightings.