Comprehensive Sensitivity Analysis of Cost-Benefit Analysis Variables for Transport Interventions

Cost-benefit analysis (CBA) is the universally applied tool to assess economic viability in assisting decisions on transport investments. Its framework is heavily influenced by the numerous variables it considers through estimating and valuing the intervention’s effects. This paper – utilising the authors’ previously implemented CBA test environment – comprehensively analyses the sensitivity of significant variables of three typical CBA models for transport interventions (road, rail and urban) to understand the prevailing appraisal approach better and to help focus on further methodological improvements. Morris and Sobol methods were selected to study the global sensitivity of and the relations between the input parameters of the models. The sensitivity test of the three analysed models provided similar results regarding which variables are most influential in CBAs. Input variables such as the investment cost, the economic discount rate, forecasted GDP changes and specific elasticities to these GDP changes often have a firm but mostly linear effect. Value of time, vehicle operating cost and mode choice-related parameters such as car availability, car occupancy rate, level of service indicators (e.g. frequency of service) and potential to induce travel demand (proxied by a ‘no travel’ parameter) are inputs with considerable linear effects and greater interactive effects.


INTRODUCTION
Transport systems still play a crucial role in our modern world, especially in cities.Despite the excessive growth of online communication and administration, the movement of people and goods on the transport infrastructure is essential for the seamless flow of everyday social and economic life.Tackling ever-present challenges in transport systems, improving and adapting them often require significant investments.As these interventions are usually not financially viable (hence provision of transport options is also a public service), public funding is needed to realise such projects.Due to the general scarcity of public funds, the efficiency of public investments needs to be guaranteed.The assessment of social viability and value for money is the most widely used way of appraising transport projects.For this purpose, cost-benefit analysis (CBA) has become a universal tool over the last few decades because of its coherent, relatively robust framework and holistic, equitable approach [1,2].Social CBAs estimate the net impact of projects from the perspective of society, thus accounting for investment and operational costs and social impacts such as the costs or benefits of induced changes in travel times, safety, vehicle operating costs and environmental burden.To measure these costs and benefits, CBA uses estimated values that society places on these effects.Based on simple performance indicators, CBA metrics can assist investment decisions.Not only can the viability of a specific project be judged, but it can also support project ranking by conferring legitimacy to contested decisions [2].Promet -Traffic&Transportation. 2024;36(4):580-592.
Transport Engineering The CBA framework is heavily influenced by the numerous variables it considers [3].On the one hand, it uses specific costs to monetise estimated impacts.On the other, impacts are usually forecasted by sophisticated transport models.Thus, the process of estimating and valuating changes caused by the analysed project is the core of a CBA calculation.Inevitably, this is where most of the inaccuracies and limitations originate.Based on [4], systematic optimism bias is also an issue, as the appraisal of public projects tends to underestimate costs and overestimate demand and benefits.There are many ongoing methodological debates on how to improve CBAs.Most of them focus on valuation issues involving surveying techniques, but discounting and handling uncertainties are also hot topics [5][6][7].However, for larger-scale, so-called transformational projects, the conventional appraisal techniques must be improved as well to account for land use changes and wider economic impacts; that is how additional effects are caused through accessibility changes [1].
Partially due to the mentioned sensitivity and the significant uncertainty, some view the results of welfare economics and CBAs as arcane and unconvincing to decision-makers.That standpoint is supported by the findings of [8]: CBA results do not really influence investment decisions.However, no alternative evidence-based approach has emerged to assist decision-making.Yet the measurement of uncertainties associated with dubious valuations and scenario assumptions in CBAs has evolved in recent years, which can help to further sophisticate CBAs [5,9,10].
A previous paper by the authors [11] provided a test environment that can experiment with possible methodological improvements for transport CBAs.Test CBA (and underlying transport) models were developed based on prevailing appraisal techniques for three typical (and hypothetical) intervention types: bypass road, interurban rail investment and a missing multimodal link in an urban transport system.This paper -utilising the authors' previous CBA test environment -aims to comprehensively analyse the sensitivity of major variables of typical CBA models for transport interventions to understand our current appraisal approach better and to help focus on further methodological improvements.The authors deliberately decided not to use a case study for application in this specific research.Adding a case study as a single reference point to understand the general implications could be misleading.That is why the authors applied the methods in three different CBA models, with different levels of detail in a controlled manner.All three models were derived from real-world case studies.
Section 2 presents the analysed input variables of transport CBAs and the methodology of their sensitivity tests.Section 3 demonstrates the results of these tests.Conclusions are discussed, and further research steps are drafted in Section 4, focusing on handling uncertainties and improving the appraisal methodology.

METHODOLOGY
The following section describes the methodology used to analyse the sensitivity of transport CBAs.The applied CBA test model and its main variables are outlined.Then the methods of the applied sensitivity tests are introduced.

Main input variables of CBAs
In order to analyse the sensitivity of the essential (input) variables of transport CBAs, first, one should explore all of the relevant variables and their connections to each other, thus, the effect mechanisms of an economic CBA for typical transport interventions.As mentioned earlier, a previous CBA test environment -created by the authors -has been applied.Details of it can be found in [11].In this chapter, a summary is provided, and the most relevant variables and the general functioning of the CBA model are described again.
The CBA test environment consists of three models representing typical transport intervention types derived from a sectorial segmentation: Bypass (road) Model, Rail Model, and Urban Model.These are based on the Hungarian CBA practice [12] that relies on the standard European CBA guidelines [13].They were developed to characterise the typical relations of input variables.The complexity of the models gradually increases in the above order to demonstrate the staggering intricacy of different types of CBAs.The Bypass and the Rail Model are spreadsheet ones, including the underlying transport models.The Urban Model combines a spreadsheet CBA calculation with a four-step transport model implemented in PTV VISUM (version 2020).
The input variables of the models were selected to include all relevant factors that arise in appraising a typical project.It was also intended to have a relative coherence of input variables between the three sample models.These variables feed the successive calculations of the transport and CBA sub-models.Behaviour modelling is based on the internal utility functions in the models, and these are intentionally unchanged for all the modelling steps within each model (unfortunately not the case for all actual appraisals).The ultimate results are the three main economic indicators usually calculated in a CBA: economic net present value (ENPV), economic rate of return (ERR), and benefit-cost ratio (BCR).Benefit calculations are based on the logsum method.In this paper's sensitivity analysis, only ENPV will be considered from these interdependent indicators.
The models use an incremental method and calculate with a 30-year-long analysis period.Investment periods and cost structures are based on industrial experience, while residual value is calculated on a remaining life-span basis.
In the simplified study area of the Bypass Model, there is a road between two cities that goes through another city.Regular bus service is also available between these cities.The analysed project is about constructing a new bypass connection and refurbishing connecting road sections.The passenger trip demand is fixed between the origin-destination pairs.There are two alternative modes of transport (private car and public bus), plus there is a 'no travel' option.Fixed freight trips of HGVs are also included.Mode choice is calculated by a standard logit model, which considers car availability.For the do-something (with investment) scenario, there is also a route choice issue (transit using the bypass or not), modelled in a logit way.Travel speeds and times depend (iteratively) on the saturation of roads.This model can represent interventions where a new link is added to the network and illustrate the effect of a quicker but usually longer alternative route.
The Rail Model is about increasing the speed of a railway line that connects two cities.There is also a road connection (with a bus service), and the road bypasses another town in between the cities.There is also an in-between rail station further away from the central town, that can be reached by a bus transfer.The passenger trip demand is fixed.There are three alternative modes of transport (private car, public bus or rail), plus a 'no travel' option.Mode choice is calculated by a standard logit model, which considers car availability.Freight transport is not included in the model.The model does not incorporate route choice decisions.Travel speeds and times depend (iteratively) on the saturation of roads, identically to the Bypass Model.There are in-built iterations to find network equilibrium as mode choice and road saturation are interdependent.In contrast to the Bypass Model, besides the disutility of travel time, travellers also consider the value of reliability (VOR), for which the average delay (lateness) and the standard deviation of travel times are used.This model can illustrate interventions where there is an investment into rail in an interurban environment with multimodal competition.
The Urban Model is about building a new bridge in a city region with a simplified radial transport network.Cordon zones are included to represent the suburban area.In the city centre, there are traffic-calmed roads.Public transport services are available for every origin-destination pair.Rail, tram and bus connections are also provided.Rail connectedness is, however, limited.The analysed investment creates a new road connection in the southern part of the city and extends a tram ring to the southwest.Transport modelling follows a conventional four-step logic.Trip generation is based on structural data (population, workplaces, amount of services).Motorisation levels of zones and freight transport are also considered.Trip distribution and mode choice are calculated with a standard logit model.There are two alternative modes of transport (private car and public transport), plus a 'no travel' option.Parking charges are applied for more congested zones, influencing mode choice decisions.Structural data and network parameters are fixed.Traffic assignments are equilibrium based for private and headway-based for public transport.Travel speeds and times Promet -Traffic&Transportation. 2024;36(4):580-592.depend (iteratively) on the saturation of roads.VOR is also included in the model, but -due to processing limitations -only for private cars through the standard deviation method.
Table 1 summarises all input variables considered in the sensitivity analysis of each model.The most apparent parameter of an economic CBA is the economic discount rate that influences the discounting of costs and benefits during the analysis period.Cost-related variables are the initial investment cost of the intervention and operational costs throughout the operation period (operating, maintenance and replacement costs).These cost elements partially consist of personnel costs, which need to be fiscally corrected in the economic analysis as certain taxes on personnel costs should not be considered as social costs (e.g.personal income tax) but transfers within the society.Thus, ratios of personnel costs and fiscal correction factors on them are also variables to be analysed.A marginal cost of public funding also needs to be considered (as a premium on all costs funded by public money), representing the price in relation to the inefficiencies incurred by raising public funds (e.g. through taxation).
Concerning the calculation of economic benefits, the impact on the transport system needs to be forecasted.For that purpose, standard transport modelling methods are applied.In those calculations, motorisation level(s) and average car occupancy determine car availability (% of who can choose to travel by car) during travel mode choice decisions.Public transport fares and levels of services (represented by the frequency of service) are also essential factors characterising the utility of public transport in making those mode choices.For private cars, vehicle operating cost (VOC) is a significant influential factor in both mode and route choice, while it also affects freight decisions.VOC is, therefore, one of the main possible benefits of a project.Similarly to income tax and personnel costs, the fuel tax rate is considered for the fiscal correction of VOC.The other variable influencing mode and route choice decisions, thus benefit calculation, is the value of (travel) time (VOT), the weighting factor in monetising journey times.Except for the Bypass Model, the value of travel time reliability (VOR) is also considered.For mode choice modelling, alternative specific constants (ASCs) are also used to represent other factors that are not endogenous variables in the transport model.The (dis)utility of the 'no travel' option is characterised only by its ASC.
Besides VOT, VOR and VOC, there are benefits from external effects such as changes in the number of accidents and the burden placed on the environment.Specific accident and environmental costs account for these impacts.Except for the Urban Model, there is also a time loss factor representing delays during construction.Lastly, GDP change and its elasticities of traffic change and specific costs (VOT, VOC, accident, environmental) influence the calculation of benefits in many ways.
The sensitivity analysis does not include the number of trips and traffic generator factors such as population and the number of workplaces or services.This was mainly due to processing limitations; however, GDP elasticity of traffic change indirectly shows that kind of sensitivity.The amount of parking charges (of the Urban Model) is also not included; still, it is not considered a sensitive variable within the whole CBA model.It was also intended to analyse crowding factors for the value of in-vehicle time for public transport services; however, also due to processing limitations, this option is not yet included in the model.

Sensitivity tests of input variables
The most basic quantitative sensitivity test is when each input parameter is changed one by one with a certain amount, and its impact on output variables is observed.It is called the one-factor-at-a-time (OAT) method, a local sensitivity test [14].In our CBA models, it was implemented as a simple 'for' loop in a VBA code, which changed each selected input parameter, then ran the model and stored the output values.With this method, input parameters that have the most significant individual effect can be identified.
There are also global sensitivity analysis methods, which provide an insight into the workings of the models and have a more complex description of the sensitivity of each analysed variable [14].For this study, Morris and Sobol methods were selected to understand the global sensitivity of and the relations between the input parameters of the models.
As mentioned, all of our models were spreadsheet ones, while the Urban Model also uses PTV VISUM for modelling transport changes.The sensitivity test itself was implemented in the R software (version 4.2.1).Thus, the first challenge was automating the sensitivity analysis process and connecting these software tools.Excel (where our models reside) was selected as a central platform, from which both R and VISUM were called via VBA macros.The experiment needed to be decoupled in R, for which the decoupling function was applied from the sensitivity package (version 1.27.0).When a sensitivity analysis method is called with no model (i.e.argument model = NULL), it generates an incomplete object x that stores the design of experiments (field X), allowing the user to launch the corresponding simulations 'by hand'.Then it will enable passing these simulation results to the incomplete object x and estimating the sensitivity measures.
So first, the experiment designs were created, which were brought to the spreadsheet models.The input variables were changed according to the designs, then models were run, and each output result was stored.Ultimately, these results were brought back to R, where the sensitivity analysis results were calculated.There were minimal challenges related to the automation of the Bypass and Rail Models, but the Urban Model with the integrated VISUM transport model proved demanding.In this case, each change in the input parameters required a new run of the transport model in VISUM.Although the applied transport model was relatively simple, it significantly increased the runtimes.
During the sensitivity tests, changes to input variables were intentionally uniform, as the purpose was to discover their sensitivity and relations with each other.Therefore, it was not intended to suppose a distorted distribution for any input variable, even if some ex-post evidence would suggest that (e.g. investment costs tend to be underestimated).
In the Morris elementary effect (EE) method, the input space is considered as a grid and trajectories are sampled among its points.The method captures output variation when one of the trajectory points is moved to one of its closest neighbours.This variation is called an elementary effect.A certain number of trajectories r are generated to observe the consequence of elementary effects anywhere in the input space.Finally, the method summarises these elementary effects to estimate global sensitivity in the output space.The Morris method is fast (needs fewer model executions, (p+1)•r) but comes at the cost of not being able to differentiate non-linearities from interactions [15,16].
To implement the Morris method, the Morris function was used from the sensitivity package of R.This function uses the following parameters: p: the number of input variables r: number of repetitions design: a specific design type, Morris's OAT design [15] was used with the parameters: 1) levels: the number of levels of the design 2) grid jump: the number of levels that are increased/decreased for computing the elementary effects 3) binf: the minimum value for the factors 4) bsup: the maximum value for the factors As a result, the Morris function produces the EE vector, which contains the elementary effects.The sensitivity measures, μ and σ, proposed by Morris, are the distribution's mean and the standard deviation.The mean (μ) assesses the overall influence of the analysed factor on the output, which is a good proxy for the magnitude of the linear effect of an input variable.The standard deviation (σ) estimates the ensemble of the factor's higher-order effects, i.e. non-linear effects and/or effects due to interactions with other factors.[16] proposed to replace the use of μ with μ * .The use of μ * solves the problem of the Type II error (failing to identify a factor of considerable influence on the model) to which the original measure μ can be exposed.Table 2 shows the parameters used in each of the Morris sensitivity tests.The other applied global sensitivity analysis method was the variance-based Sobol method.This method works with a probabilistic framework to calculate the variance of the output, which can be attributed to sets of inputs.It can deal with non-linear responses and measure the effect of interactions in non-additive systems.
Sobol indices are derived from a functional decomposition of the output variance.First order index (also known as 'main effect') measures the output's variance caused by a given input variable.The so-called 'total effect' index suggested by [17] adds additional (interaction) effects caused by a given input variable to the main effect.The estimation is based on a Monte Carlo sample of mutually independent input variables, in which standard errors are derived by bootstrapping [18].
To implement the Sobol method, the sobolmartinez function was used from the sensitivity package of R. It implements the Monte Carlo estimation of Sobol's indices for both first-order and total indices using correlation coefficients-based formulas, at a total cost of (p+2)•n model evaluations.These are called the Martinez estimators.This function uses the following parameters: p: the number of input variables -X1: the first random sample with n values created based on the unified distribution between 'low' and 'high' values -X2: the second random sample with n values created based on the unified distribution between 'low' and 'high' values nboot: the number of bootstraps calculated by the theoretical formulas based on confidence interfaces of correlation coefficient [19].
-The analysis provides the following results: -S: the estimations of Sobol's first-order index, i.e. the main effect index.It measures the effect of varying a given input variable alone but averaged over variations in other input parameters.It is standardised by the total variance to provide a fractional contribution.-T: the estimations of Sobol's total sensitivity index, i.e. the total effect index.This measures the contribution to the output variance of an input variable, including all variance caused by its interactions, of any order, with any other input variables.
Table 3 shows the parameters used in each of the Sobol sensitivity tests.

RESULTS
The following chapter describes the results of the sensitivity tests of transport CBAs.Results are illustrated for three typical types of CBA: road projects (Bypass Model), rail projects (Rail Model), and urban projects (Urban Model).

Sensitivity results of the Bypass Model
The simplest model in the introduced test environment is the Bypass Model.Its OAT local sensitivity test showed those input parameters that have the most considerable influence on the output in case of a standard +1% change.For the Bypass Model, the most sensitive inputs are car availability (motorisation, +10.1% change in the output), VOT (+7.0%) and VOC (-6.9%).
The Morris method provided further insight into the global sensitivity of input variables, which is visualised by Figure 1.The estimation confirmed that, based on their linear effects (μ * ), car availability, VOC and VOT are the most significant variables.However, the estimation of interaction effects (σ) suggests two things.First, most input variables have a far greater linear effect than interaction effect.The exception from the more influential inputs is the average car occupancy factor, which has a slightly larger interaction effect.Second, those input variables attributed with the strongest linear influence have relatively small interaction effects.The Sobol method, illustrated by Figures 2 and 3, shows a similar overall result: car availability, VOT and VOC are the most influential factors in the model.There is no significant difference between the main and total effects, thus suggesting that interaction effects tend to be smaller in this model setup.However, compared with the results obtained by the Morris method, the average car occupancy factor seems to be a far less significant input variable, while no travel ASC appears to be a far more significant one.

Sensitivity results of the Rail Model
In the case of the Rail Model, the local sensitivity test concluded that the investment cost (-3.5%), the economic discount rate (-3.5%) and the average car occupancy rate (-3.2%) are the most sensitive input variables.However, VOC (+2.2%),VOT (+1.9%) and GDP change (+1.8%) also appear to be relatively sensitive parameters.
The global sensitivity analysis based on the Morris method (see Figure 4) showed that the linear effects of the average car occupancy rate, VOC and the economic discount rate are the most significant.Investment cost, VOT, 'no travel' ASC, the number of trains (service frequency) and GDP change are also influential inputs.Contrary to the Bypass Model, more input variables have a considerable interaction effect.For instance, 'no travel' ASC and VOT have a greater interaction effect than linear.Car availability, VOR and train fare are close to the 45-degree line indicating comparable linear and interaction effects.The Sobol method (see Figures 5 and 6) provides a similar result.In the case of the main effect, the average car occupancy factor, VOC and the economic discount rate are the most influential input variables.Based on the total effects, the average car occupancy factor and VOC still have the strongest influence, but the sensitivity of VOT, 'no travel' ASC and the number of trains emerge.The economic discount rate has a considerable interaction effect, so its overall effect is also substantial.These findings are coherent with the metrics from the Morris method.
Calculations based on the Morris method suggest that most of the inputs have comparable linear and interaction effects.From these, 'no travel' ASC, VOT, VOC, car availability and the average car occupancy rate are the most significant, respectively.The first two have a much stronger influence on the output.Economic discount rate, GDP change, GDP elasticities and investment cost have a considerable linear effect, but their interactive effects are limited.VOR appears to be a quite interactive parameter, but its overall influence is moderate (see Figure 7).
Sobol indices, again, show a fairly different picture to the Morris method.Some of the more sensitive inputs from previous testing do not seem to be that sensitive based on the first-order indices (e.g.VOT), see Figure 8.Moreover, the dispersion of the main effect (see Figure 9) is relatively large, larger than in the other models.Similarly to the Rail Model, total effect indices suggest significant differences in the inputs' interactive effects.'No travel' ASC, VOT, VOC, car availability and the average car occupancy rate all have considerable interactivity, which is in line with the findings of the Morris sensitivity screening.

CONCLUSIONS
The sensitivity test of the three analysed models provided similar results regarding which variables tend to be the most influential in CBA calculations.Input variables such as the investment cost, the economic discount rate, forecasted GDP changes and specific elasticities to these GDP changes often have a firm but mostly linear effect.VOT, VOC and mode choice-related parameters such as car availability, car occupancy rate, level of service indicators (e.g.frequency of service) and potential to induce travel demand (proxied by the 'no travel' ASC parameter) are inputs with considerable linear effects and greater interactive effects.
The uncertainties of input variables could be handled in multiple ways.A usual procedure is to add 'premium' values or multipliers based on historical data and ex-post experience to balance previously observed systematic biases.However, the heart and soul of these appraisals is the modelling of travel behaviour, forecasting changes and monetising them.More and more accurate surveying techniques and gathering of ex-post evidence are crucial practices to improve the accuracy of appraisals and the trust in them [1,2,20].Quantitative risk assessments can also help to identify sensitive spots of the appraisals and to put results into an adequate context.
From a sensitivity perspective, parameters concerning trip generation (and transport and land-use interaction where applicable) should be further analysed as our results on car availability and potential of induced travel (through the 'no travel' ASC) suggest that this can be a crucial part of forecasting future travel Promet -Traffic&Transportation. 2024;36(4):580-592.

Figure 1 -Figure 3 -
Figure 1 -Sensitivity screening of the Bypass Model with the Morris method

Figure 4 -
Figure 4 -Sensitivity screening of the Rail Model with the Morris method

Figure 6 -
Figure 6 -Total effect Sobol index estimates for the Rail Model

Figure 7 -
Figure 7 -Sensitivity screening of the Urban Model with the Morris method

Figure 8 -
Figure 8 -First-order Sobol index estimates for the Urban Model

Figure 9 -
Figure 9 -Total effect Sobol index estimates for the Urban Model

Table 1 -
Considered input variables in the sensitivity tests

Table 2 -
Applied parameters for Morris sensitivity tests

Table 3 -
Applied parameters for Sobol sensitivity tests