Evidence-based Improvement Policies in Public Management : A Case Study of Public Projects in Barrancabermeja City , Colombia

Studies show that many organisations face a deficient capability of information analysis to make decisions effectively, due to a lack of statistical evidence and the poor quality of data, among other factors. This has become an increasing problem in some public organisations of developing countries. This study proposes a methodical approach for the policy formulation improvement, based on statistical evidence. Conducted as a case study in the city of Barrancabermeja in Colombia, this work identified relevant factors that influence the approval procedure, tested as statistical evidence at a confidence level α = .05. The sample consisted of 55 civil engineering projects completed during 2008–2013. The statistical evidence found in this study suggest a benchmark of 15 days as the expected duration for the registration and approval processes of projects, which may be adopted as an assessment policy for public project management in Colombia. Additionally, the study provided statistical evidence to formulate policy related with: the memorandum of commitment to the community benefited by the project, the methodology for the registration of public projects, and the project budget estimation. Thus, this study advocates the use of statistical evidence as an appropriate strategy for the improvement policy formulation of project management processes and decision making in public organisations.


INTRODUCTION
One of the objectives of public management is to guarantee the best use of public resources to create maximum benefit for citizens.A public investment project (PIP) is a government initiative to create, expand, improve, or restore the production or supply of goods or services, through the use of the public resources of the country.In Colombia, PIPs are funded mainly by official entities at the national and local levels, and executed by construction contractors.Most of these PIPs are civil engineering projects developed through the traditional Design-Bid-Build project delivery method.PIPs are registered and systematised by means of a national public registry (BPIN in Spanish).The BPIN is an instrument that facilitates the monitoring of the government's development plans and supports the processes of planning, scheduling, monitoring, control, and assessment of formulated PIPs within the public sector (DPN, 2011a).
To systematise a PIP in the BPIN, the project must be formulated, approved, and registered in accordance with the guidelines established in the Adjusted General Methodology (MGA in Spanish) by the National Department of Planning (DNP in Spanish).The MGA is a management tool that aims to make the process of formulating, scheduling, and evaluating PIP quick and efficient.The MGA guides the collection of information about a project, such as relevant needs, objectives, and alternative solutions.Once the project is identified, registered, and scheduled through the MGA, a Basic Investment Statistics sheet (EBI in Spanish) is generated (DPN, 2011b).The EBI is a summary sheet, which must be filed in the BPIN and contains all the basic information concerning the project.
Each PIP must be formulated and presented to the municipal planning offices, where the requirements established in the MGA are verified and its specific characteristics such as technical, financial, environmental, and social aspects are reviewed and approved.Thereafter, the project must be registered in the BPIN and assigned to the appropriate municipal office for management.In Colombia, the assessment of public management focuses on the efficiency and effectiveness of its processes (DAFP, 2012); therefore, assessing the PIP management process is of interest to most public entities.Nevertheless, some municipal administration offices face a deficient capability of information analysis that they need to account for their performance assessment.
This capability shortfall is an increasing problem in many organisations.A survey carried out with 254 managers in US companies, whose revenues total more than USD 500 million, found that 40% of managers base their decisions on qualitative and subjective criteria rather than quantitative analysis due to the shortage of reliable data.Furthermore, it was found that many employees of large firms lack the appropriate quantitative and analytical skills to make decisions objectively.Therefore, 72% of the companies surveyed were trying to improve their analytical capability, shifting from traditional methods based on subjective perceptions to evidence-based methods (Accenture, 2008).In public management, some concerns arise related to the quality of the evidence collected for decision making as well.This is the case of the administration of Barrancabermeja City, Colombia, which faces a deficient capability of information analysis to make decisions effectively, due to a lack of statistical evidence and the poor quality of data, among other factors.This has become an increasing problem because of scarce publications and research with practical solutions for the formulation of improvement policies in public organisations.To this end, this study proposes a statistical evidence-based methodology that will help the city administration to improve its public project management processes through the formulation of improvement policies.

Description of the Problem
The city of Barrancabermeja is located in western Santander, Colombia, on the banks of the Magdalena River.It has more than 200,000 residents and a total area of 1,348 km 2 , of which 30 km 2 (2.2%) is urban and 1,318 km 2 (97.8%) is rural.The city is comprised of 154 neighbourhoods organised into 13 zones.At present, Barrancabermeja manages a public budget of USD 100 million and a 2012-2015 investment plan of USD 300,000 (Alcadía de Barrancabermeja, 2012).
The administration of Barrancabermeja approves, registers, and manages proposed PIPs through the Municipal Planning Office (MPO).According to the national Policy of Good Governance (DPN, 2014), every local administration should be accountable for the results of its management, and therefore, must record all information referring to the proposed PIPs.In compliance with Law 872 of 2003, Barrancabermeja has adopted three guidelines: the Integrated Quality Management System, the Internal Control Standard Model, and the Public Management Technical Quality Standard, to promote alignment of public administration accountability with the national guidelines.
A PIP can be formulated by the community, contractors, or any public organisation, according to the MGA guidelines.In Barrancabermeja, this PIP must be registered by the MPO, which verifies its compliance with the requirements and issues the approval.The MPO has 5 business days to review the project and send it to the corresponding technical office.Then, the technical office has 8 business days to verify whether the project meets the basic approval requirements, and 10 additional business days to approve the technical and social requirements.Finally, the technical office has 5 business days to determine the viability of the project, and another 5 days to register it in the BPIN.In total, the entire approval process provides for 33 business days to officially approve and register the project.However, the Municipal Administration of Barrancabermeja currently lacks a formal system for the formulation, approval, and registration of PIP.
In addition to the lack of a formal system for PIP management, when assessing the performance of public management in the Municipal Administration of Barrancabermeja, some concerns arise over the quality of evidence collected.Is the data used in the statistical analysis process sufficient to establish cause and effect relationships and to design improvement plans?Can the performance of public management be inferred from data of the quality available?What methodology would guarantee that the results and conclusions of the study are not a consequence of random chance?This study proposes a methodical approach based on statistical evidence of relevant influencing factors of project management processes to meet this need, and thus, to formulate management improvement policies.

Assessment of Public management
The assessment of public management performance is based on a systematic measurement and comparison of results to improve the decision-making process (DAFP, 2012).The management of PIP can be assessed with respect to three phases of the project life cycle: first, the formulation and planning of the project is assessed during the feasibility and design phases; second, the use of project resources is evaluated during the implementation phase; and third, the impact or benefit of the project is assessed during the operation phase.In Colombia, the Administrative Department of the Public Function (DAFP in Spanish) is the national entity responsible for establishing policies and guidelines to assess the efficiency of public management.
In the assessment of public management, performance indices are instruments that provide evidence to formulate improvement and resource allocation policies.Performance indices are derived from statistical information within a process control system.They support the processes of problem area identification, of planning, and of control.According to the DAFP, an indicator is a quantitative or qualitative observable measure that describes a feature of reality by means of a variable.This study used management performance indices as statistical evidence to identify problems in the PIP approval process, and thus to formulate management improvement policies.Specifically, indices were used as a measurement and baseline tool to benchmark the time required to register and certify PIP in the MPO.

Evidence-based Improvement Policies
Evidence-based Management (EBM) is a methodology that scientifically determines the most reliable, valid, and relevant information (evidence) to support the decision-making process (Marr, 2009;Barends, Rousseau and Briner, 2014;Pariente Fragos, 2009).EBM arose from evidence-based medicine, which utilises the scientific method to collect data, and to formulate and test hypotheses statistically to solve a problem or improve medical practices (García del Junco and Casanueva Rocha, 2010; Marr, 2009).Similarly, EBM seeks to support the management decision-making process with the best available scientific evidence to ensure a reliable decision (Pariente Fragos, 2009).EBM is comprised of five steps: (1) definition of objectives or problem; (2) collection of data through quantitative or qualitative methods; (3) statistical data analysis; (4) presentation of results; and (5) decision making and impact assessment (García del Junco and Casanueva Rocha, 2010;Marr, 2009;Pariente Fragos, 2009).
In the same vein, Evidence-based Policy (EBP) is an extension of evidencebased medicine, and helps policymakers to make decisions about programmes, projects, or practices that are supported by high-quality and systematic evidence collected from experiments (Suttcliffe and Court, 2006).EBP promotes effectiveness in public institution performance.Careful evaluation of the data is essential for a successful implementation of EBP.A policy is a statement that guides action to solve an issue.Therefore, its formulation and impact assessment require the use of accurate data, quantified and analysed statistically (Davies, Nutley and Smith, 2000).
This data can be organisational or scientific (Barends, Rousseau and Briner, 2014).Organisational evidence is comprised of data, figures, and facts collected by the organisation, while scientific evidence refers to the information derived from observations or experiments to support or refute statistically a stated hypothesis.Evidence is analysed through quantitative and qualitative methods to propose and test models that attempt to predict, simulate, and explain the situations that could be experienced by the company to make correct decisions (Marr, 2009).

Aim and Scope of the Study
This study proposes a methodical approach for the improvement policy formulation in public organisations, based on statistical evidence of project management processes.The study is an example of an applied research conducted as a case study in the public administration of Barrancabermeja City, Colombia.The work identified the relevant project information requirements that influence the approval procedure for public projects in Colombia, which were then used as evidence for the improvement policy formulation of project management processes in the administration of Barrancabermeja.In addition, this work proposed a predictive model of approval success, with the significant statistical evidence as predictors, which will allow the identification of appropriate control actions.

METHODOLOGY Research Design and Sample
This study utilised a cross-sectional research design with cross-tabulation and logistic regression analyses to identify the most influential factors in the PIP approval procedure.The project information was taken from the Barrancabermeja MPO database and the analysed projects were managed through the Municipal Infrastructure Office.
A cross-sectional design has three features: no time dimension, identification of differences rather than change over time, and groups classified by existing differences rather than random allocation.As such, the cross-sectional design can only analyse differences and significance between factor-groups (De Vaus, 2001).
No time dimension means that data are collected and analysed at one point of time, as a whole.Cross-tabulation technique provides a simple way to analyse the responses of groups in a sample and test if two factors or variables are statistically significant.In other words, cross-tabulation technique is as a series of related frequency tables combined into one table (De Vaus, 2002).Finally, the logistic regression is a prediction model for causal analysis of dichotomous dependent variables, analysis of success or fail cases.
This study used a convenience sample of 55 independent PIPs executed during 2008-2013, whose information met all requirements (see next paragraph).This study used a convenience sampling because of the availability of the total of projects completed during 2008-2013 to be analysed.This is a specific type of nonprobability sampling method that relies on data collection from the population's projects, which are conveniently available for the study.The total cost of the sampled projects was approximately USD 200 million with an average cost per project of USD 4 million (SD = 11.02) and an average duration of 7.8 months (SD = 4.8).

Variables and Statistical Techniques of Analysis
The study analysed the time elapsed to approve each project, which was classified by year, type, and cost.The time to approve (TC) was defined as the time elapsed, in calendar days, between the dates at which the project was received and approved by the MPO.
Non-parametric techniques such as contingency tables and logistic regression models were used to evaluate statistical evidence of correlation between the dependent variable (TC) and its influencing factors.Specifically, contingency tables tested the hypothesis that a given specific factor influences the time required to approve a project, while the logistic regression model tested the hypothesis that a specific factor predicts the likelihood that the time to certify a project will be greater than 15 days.The analyses were performed at a level of significance of α = .05using the statistical software package SPSS™, v. 21.
The contingency tables were used to calculate the odds ratio (OR) and the relative ratio (RR), which are risk measures.The OR indicated the influence of a given specific factor upon the probability of a project being processed within a reference time, while the RR indicated the probability of a delay caused by an influencing factor.Both ratios were considered statistically significant when their values were greater than 1.Finally, the results of the study were discussed and validated with the professionals responsible for the management processes of the Barrancabermeja MPO.

RESULTS
This section first presents the results obtained when testing the correlation between each influencing factor and the time required for project approval, followed by the significant factors that affect the approval process and their corresponding ratios understood as risk measures regarding non-compliance to the requirements.Finally, this section specifies the factors that can be considered as predictors of project approval time.

Evidence of Correlation
Projects were classified into five groups based on the time required to approve them (Table 2).Of the sample, 47% of PIPs were approved in 15 days or less (X ¯ = 8.1, SD = 3.2; N = 55), which preliminarily suggests that 15 days could be set as a benchmark to measure the performance of the PIP approval process.First, the study tested the correlation between the time for approval and the year of project registration (Table 3).With this test, the study sought statistical evidence to establish whether the project's year of registration was an influential factor on the time of approval.For the cross-tabulation analysis, the variables TC and year of registration were split into groups; TC ≤ 15 days and TC >15 days for the former, and 2008-2010 and 2011-2013 for the latter.The proportion of projects with an approval time of ≤ 15 days was 47%.The Chi-square test showed insufficient statistical evidence for correlation between the approval time and the year of registration (χ 2 = 0.36; df = 1, N = 55; p = 0.55).Table 3 provides information regarding the number of projects registered by year.These projects corresponded to two different government administration periods.During the first period, 2008 to 2011, 24 projects were executed: an average of 6 per year.During the second period, from 2011 to 2012, 31 projects were executed: an average of 15.5 per year.This suggests that the MPO of Barrancabermeja should pay increased attention to the PIP management process, since it is probable that the number of registered projects will increase over time.Therefore, it is necessary to formulate policies that improve the current registration process of public projects in this administration.
Thereafter, the study tested the correlation between the TC and the type of project (Table 4).Water supply and sewer, and infrastructure were the most common types of project in the sample.This finding is in keeping with the Development Plan of Barrancabermeja 2012-2016, which prioritises investment efforts for these two.For statistical hypothesis testing, the variable type of project was split into three subcategories: water supply and sewer, infrastructure, and other.The Chi-square 62/PENERBIT UNIVERSITI SAINS MALAYSIA test found insufficient statistical evidence to show correlation between type of project and time of approval ( χ 2 = 0.13; df = 1, N = 55; p = 0.94).Finally, the study tested for correlation between the TC and project size (total budget).The Chi-square test showed sufficient statistical evidence for correlation between these variables ( χ 2 = 5.2; df = 1, N = 55; p = 0.02).Most projects (63%) with a budget greater than USD 1 million had a time of approval above 15 days, while 37% of projects with a budget below USD 1 million were certified in 15 days or less.

Significant Influencing Factors on Approval Time
Table 5 shows the proportion of projects whose approval time exceeded 15 days, based on whether the project complied with various approval requirements (only requirements with significant correlation at p < 0.1 are shown).However, their level of influence differs.The requirements R3 (memorandum of commitment to the community), R5 (EBI sheet), R6 (MGA), and R7 (schedule) showed a high influence; more than 70% of projects that did not comply with these requirements exceeded 15 days.Meanwhile, R8 (budget) and R11 (digital record of information) showed a moderate (67%) and low influence (44%), respectively.These results were used as evidence to formulate management improvement policies.The study calculated the risk of increasing TC that is associated with noncompliance to the requirements, through the OR and the RR (see Table 6).R7 (schedule) was the factor with the highest risk.The proportion of projects that did not comply with requirement R7 and had a TC > 15 days was 6.8 times greater than those that complied (OR = 6.8, 95% CI = 1.9-24.7).Using the RR, projects that did not comply with requirement R7 were 2.2 times more susceptible to be registered after 15 days (RR = 2.2, 95% CI = 1.3-3.5).Similarly, R3 (memorandum of commitment to the community), R5 (EBI sheet), and R7 (schedule) showed sufficient statistical evidence to be considered requirements with a high level of risk.The study did not find sufficient statistical evidence to associate any of the economic and environmental requirements with an altered level of risk.

Statistical Evidence for Significant Predictors
A logistic regression model, at a significance level α = .05,allowed for the identification of those requirements associated with the likelihood that the project approval process would have a TC > 15 days.The model indicated that the documentation requirements R3 (OR = 19.3),R9 (OR = 15.4),R10 (OR = 10.9), and R11 (OR = 0.054) showed sufficient statistical evidence to be considered as predictor variables of TC > 15 days (p < .05).The technical requirement R8 (OR = 13.3) and the economic and environmental requirement R19 (OR = 11.1) were also found to be statistically significant predictors.Although the factor R11 contributes significantly to the prediction of TC, its OR and RR were statistically insignificant, so it does not represent a significant risk.The suggested model classified 86% of the cases correctly (Eq.1).

Statistical Evidence-based Assessment
The assessment of public management should be based on statistical evidence; nonetheless, this approach has often been overlooked in Colombia.Studies show that there is an improvement in the public management process when monitoring and control methodologies are implemented based on statistical evidence.
For instance, a case study in one Colombian city reported an improvement in its management indicators when a quality management system (ISO 9001) was implemented (Báez Mancera, 2013).Another case study showed how statistical evidence was used to successfully assess the performance of a city's financial public management.The study also identified relevant and significant performance indicators and their influencing factors (López, Gonzalez and Tobón, 2012).
The present study proposes the use of statistical evidence for the improvement policy formulation of public management in the Barrancabermeja MPO.Specifically, statistical evidence coming from statistical analysis should support any decision made.Currently, there is no explicit and systematic methodology for assessing PIP, nor clear benchmarks for the TC of project registration in this administration.Based on the results of this study, and taking into account the administrative environment of Barrancabermeja, this work suggests a benchmark project registration time of 15 days as an efficiency indicator of PIP management.Thus, a criterion can be established to measure the performance of future public management.
In addition to the TC benchmark, this work identified the most critical requirements to be considered for the formulation of improvement policy for the public project registration process within the Barrancabermeja administration.The requirements that most influenced TC were: memorandum of commitment to the community, EBI sheet, MGA, schedule, and detailed budget.These findings align with those of other studies.In Colombia, some studies have assessed MGA implementation and have found that the methodology is still unfamiliar and confusing (Ulloa and Castro, 2007;Lizargo Ortega, 2005).

Statistical Evidence-based Improvement Policy
As noted previously, the formulation of management improvement policy should be based on statistical evidence.In that sense, and based on the study findings, this paper suggests that the municipal administration of Barrancabermeja should set a policy of monitoring and control for the formulation and management of PIPs to help avoid the factors that can drive up the time required to approve and register projects.This recommendation is made more urgent by the fact that the average of number of projects registered increased notably during 2012-2013, and this trend seems likely to continue.
As improvement policies for PIP management, the study recommends special attention be paid to projects with budgets greater than USD 1 million.Additionally, it is suggested that the municipal administration should set a policy that supports water supply and sewer, and infrastructure projects, since these two types of project are prioritised in the municipal Development Plan for 2012-2016.Finally, to mitigate or even eliminate problems with the registration requirements of projects, it is proposed that the administration should plan and carry out workshops with the community to provide training and raise awareness about the use of these documents in the formulation of PIP.
Following a discussion of these findings, representatives of the Barrancabermeja MPO have undertaken to design a programme for community leaders to train them in the preparation of the MGA and EBI sheets for the formulation of PIP.Additionally, the MPO plans to train its staff and those in the various sectoral offices in MGA methodology.Finally, to improve firms' information about project budgets, the MPO will create a list of standardised activities and prices.Professionals with knowledge in the field must carry out this task.

CONCLUSION
Statistical evidence is understood as information supported by the results of statistical analysis.This study shows how statistical analysis can be used to provide information in decision-making processes.The results of cross-tabulation analysis with contingency tables supported the formulation of improvement policy for the PIP approval procedure and decision-making in the Municipal Infrastructure Office of Barrancabermeja.This approach allows municipal planning offices to set key performance indicators and to identify factors that impact the PIP management process.Benchmark indicators of management performance enable the identification of possible disruptive factors to formulate improvement policies that promote and guarantee the best use of public resources.
The findings of this case study contribute to the body of knowledge in public management in developing countries, using a statistical evidence-based methodology and cross-tabulation analysis that support decisions of public project offices.Thus, evidence-based improvement policies emerge as a methodological alternative that takes full advantage of the available information for the benefit of organisations.The study recommends the implementation of this methodology in Colombian MPOs to help policy makers in future decisions.
Although evidence-based approach for the formulation of improvement policy brings on some advantages, there are streams of authors that criticise this approach due to some limitations and misperceptions such as shortage of coherence and consistence of statistical evidence, quality of data and information processing, and analyst's skills (Straus and McAlister, 2000).On the other hand, Greenhalgh and Russell (2009) argues that the nature of policymaking process goes beyond "knowledge translation" or "getting evidence into practice".Statistical evidence informs policy judgements but itself does not resolve the pivotal question of what to do.In consequence, there is a long way to go on research and experience over the formulation of policy based on evidence for public management.This study advocates for this approach, which could bring many advantages to public administration in Colombia.

Table 1 .
Table 1 shows the PIP average cost, sorted by type.Water and sewer, and infrastructure projects represented 73% of the sample (40 out of 55) by count and 82% (USD 190 million) by cost.Project Average Cost by Type (USD Million) Note: *COP 3000 = USD 1 approx.Guillermo Mejía and Diana Marcela Franco-Duran60/PENERBIT UNIVERSITI SAINS MALAYSIA

Table 5 .
Statistical Evidence of Influence; Projects with TC > 15 Days Note: p* -statistical significance α = 0.05 or α = 0.10Statistical Evidence-based Risk Evidence-based Improvement Policies: A Case Study in Colombia PENERBIT UNIVERSITI SAINS MALAYSIA/63

Table 6 .
Evidence-based Risk (OR and RR); Projects with TC > 15 Days