Evaluation of Project Portfolio Management Performance: Long and Short-Term Perspective

: The complexity of project portfolio management /PPM/ is widely recognized in the specialized literature as well as its multidimensional link to the measurement of its success. In light of the variety of portfolio administration issues thoroughly studied, there is not any comprehensive model developed for the evaluation of PPM performance. Here we agree on the basic proposition that PPM success should not be evaluated by any simple consolidated result derived as a sum of the individual performance of each project. The paper presents selected results from a study of PPM success in 184 Bulgarian project-oriented organizations where several indicators have been experimentally measured. It provides insights on the measurement of project portfolio management success in both short and long-term perspective. The long-term perspective is captured by two dimensions of the strategic alignment – the first one relates to the achievement of strategic goals of the organization through selection and realization of projects. The second one is focused on the strategic resource allocation. Short-term perspective is measured by the level of projects’ success accounting for three dimensions – meeting the project’s budget, schedule, and quality of delivery. These dimensions reflect the efficiency of organizational resources utilization which can be achieved through high degree of coordination of project planning, execution, monitoring and control within an operational system of project portfolio management. Here we assume that successful governance should lead to a better alignment of projects in the portfolio with the organizational strategy. Initial findings about these dimensions of PPM success are reported and discussed in the paper.


Introduction
Project management proved to be effective for enhancing the competitiveness of business organizations in light of the substantial expansion of project-based activities worldwide.Together with the enlargement of project-based funding, many organizations have chosen to transform their structures into project-oriented ones.At the same time, practices of inappropriate selection, ineffective prioritization or suboptimal balancing of projects are frequently met which reflects into unsuccessful performance of project execution and under-delivery of project goals (Cooper et al., 2000).In many organizations even a limited number of projects are hardly accomplished as expected due to ineffective implementation of project methodologies, inadequate project management structure, or lack of project coordination (Kendall & Rollins, 2003).As a result, inefficient resource allocation and related losses as well as misalignment of project deliveries and strategic goals of organizations have been identified in project management practices.
In response to such challenges in both managerial theory and practice a specific approach to project management has been developed, namely project portfolio management (PPM), which gained increasing popularity and found a wider implementation in organizations of various size and industry (Charvat, 2003).This managerial approach is based on principles of coordinated selection, prioritization, balancing, and control over the execution of multiple projects in a project-based organization (Alexandrova, 2020).The projects' goals should be aligned with the organizational strategy matching the long-term perspectives for organizational development.The need for additional research on particular issues related to PPM performance evaluation is considered as a general deficiency especially for organizations implementing the project portfolio management approach (Kleinschmidt et al., 2007).
The anticipation of enhanced PPM performance is related to the role of project portfolio manager and its interaction with other levels of organizational management (Jonas, 2010).
One possible justification of PPM success originates from this core role, i.e. a significant impact of management involvement, potentially in both aspects (positive and negative), on PPM performance.This way, the task of boosting PPM performance is associated to transforming the formal role of project portfolio manager into real authority, leadership, and strategic involvement.Furthermore, the projects in a portfolio compete for scarce organizational resources requiring committed top management intervention for efficient resource allocation through prioritization and balancing of the organizational project portfolio.This way the enhancement of PPM performance can be facilitated by effective portfolio management in conditions of time and resource limitations (Cooper et al., 1997).
PPM outperforms the traditional framework and mechanisms of project management due to the substantial changes required for an effective implementation of portfolio structures leading to complex synergetic effects on organizational performance.In a turbulent business environment and globalized economy PPM approach has gained an increasing popularity worldwide as well as in Bulgarian project-oriented organizations (Alexandrova et al., 2015).In a multi-project environment this approach contributes to the enhancement of competitiveness by maximizing the value added by separate projects along with minimizing the costs and risks associated to their execution.This way, PPM has proved to be a source of competitive advantage for the implementing organization by boosting the performance of its project portfolio.
The aim of this paper is to provide insights on the specifics of PPM evaluation in both short and long-term perspective.The long-term perspective is captured by two dimensions of the strategic alignment -the first one relates to the achievement of strategic goals of the organization through selection and realization of projects.The second one is focused on the strategic resource allocation.The short-term perspective is assessed by the level of projects' success accounting for three dimensions -meeting the project's budget, schedule, and quality of delivery.These dimensions reflect the efficiency of organizational resources utilization which can be achieved through high degree of coordination of project planning, execution, monitoring, and control within an PPM system.Here we assume that the enhancement of PPM performance should be influenced by a better alignment of projects in the portfolio with the organizational strategy.The paper presents selected results from a study of PPM success in 184 Bulgarian project-oriented organizations where several indicators have been experimentally measured.

Literature Review
Currently, expanding number of companies which simultaneously execute multiple projects systematically develop their capacity for implementation of project portfolio management approach.This approach turns to be a key strategic instrument for gaining competitiveness and overall business success (Dietrich & Lehtonen 2005).Although initially project portfolio management has been focused on the selection of projects based on the link between project risks and benefits, nowadays it is associated to a much wider range of aspects, e.g.project monitoring and control, project prioritization, portfolio balancing, risk evaluation and management, reporting of portfolio results (Blichfeldt & Eskerod, 2008).
The definition for PPM adopted in the current study follows the one developed by Blichfeldt and Eskerod (2008) that emphasizes on managerial activities focused on "the initial screening, selection and prioritization of project proposals, the concurrent reprioritization of projects in the portfolio, and the allocation and reallocation of resources to projects according to priority".
Furthermore, the concept of Thiry (2006) is also supported in the study according to which PPM is based on resource analysis and allocation within the organization directed to achievement of organizational goals and thus maximizing the stakeholders' value.The development of PPM approach its performance evaluations is based on a variety of aspects (De Reyck et al., 2005) e.g.: • centralization and formalization of the rules and procedures for project execution; • portfolio optimization through provision of support by standardized processes, forms, and software tools; • financial and risk analyses at both levels: individual project and overall portfolio; • accounting for the constraints on resources shared by projects; • emphasizing on assessment during selection and prioritization of projects; • assuring the quality of governance at both levels: individual project and portfolio.
Following this framework, various issues related to mechanisms for PPM implementation and evaluation have been identified, e.g.assessing the scope of project portfolio, ranking of values and benefits, appraising uncertainty and risks of projects and project portfolios, etc. (Levine, 2005).The task of attaining effective PPM appears to be a significant challenge to many project-oriented organizations as far as its content and specifics are still subtle for project portfolio managers.Often the effectiveness of processes related to PPM, e.g.project selection, prioritization, assessment, etc., is not precisely defined and frequently is just generally stipulated as "doing right projects" (Cooper et al., 1998).An important direction in this respect is the development of concepts, instruments, and tools specifically designed for evaluating PPM performance and its success.
Conceptually, the current study reflects a variety of suggestions about the dimensions of PPM performance, considered in a wider prospect, as well as PPM success in a narrow sense of project portfolio goals achievement.The latter is based on the premise that the success of single projects is a necessary but not sufficient condition for the overall PPM success (Martinsuo & Lehtonen, 2007).Different elements and factors have been identified as influencing the portfolio success, among which project performance during the execution hase and future awareness of PPM (Shenhar, 2001); single project success, in respect of time, budget, and quality of delivery (Atkinson, 1999); alignment of the portfolio to the business strategy (Cooper et al., 2002); emerging synergy between projects within the portfolio (Meskendahl, 2010); portfolio balance regarding the organizational capabilities and resources (Teller et al., 2014), etc.
According to Jonas (2010), a framework for evaluation of PPM success must be "capable to capture" PPM performance as a whole, using a set of appropriately postulated success criteria related to each other in a "causal chain relationship".From this point of view, the analysis of PPM success is integrated into a complex overall assessment system at organizational level that takes into account three dimensions: process effectiveness, portfolio success, and portfolio-related corporate success.Such ideas have been considered by Shenhar et al. (2001) stating that the assessment of projects' success needs to reflect their performance during the execution and the achievement of the targeted results.Drawing on this premise, these authors derive two dimensions at portfolio level: business success (current market success and commercial performance) and corporate success (long-term economic effects regarding the portfolio).
Other authors focus on more specific indicators for PPM performance: project teams autonomy, human resource qualifications, shared knowledge and information, top management support, project stakeholders integration, etc. (Kleinschmidt et al., 2007).
However, in search of constant improvement of single project performance, a risk originating from the rivalry between projects for key organizational resources may expel the gains for any single project leading to deterioration of the overall PPM performance.For example, a recent empirical study utilizing data from 181 project portfolios provides evidence for the positive effect of project portfolio information systems for achieving PPM quality and success.This study confirms the moderating effects of formalization processes at single project, portfolio, and risk management for achieving the benefits of information systems for PPM (Kock et al., 2020).Another recent study pays attention to project performance analysis that is typically based on core indicators for project success, also termed as "project health analysis".In line with this, a "healthy portfolio" cannot perform adequately if risk management is ineffective and PPM performance reflects directly the success of business operations of project-oriented organizations (Micán et al., 2020).
One of the major dimensions of PPM performance is defined as "average project success" and typically captures the achievements of traditional performance criteria: budget, schedule and quality -the so called "iron triangle" of project management (Atkinson, 1999).
For the goals of the current study these criteria are assessed in a short-term perspective.
Besides, the synergy due to coordinated management of projects in a portfolio adds value to the summary benefits from the individual projects.The other major dimension of PPM performance is the achievement of strategic fit of the portfolio that indicates the extent to which the project goals reflect the organizational strategy (Unger et al., 2012).Such a strategic reflection is considered as a long-term perspective that contributes not only to the alignment of project goals but also to the optimal allocation of organizational resources.

Data Source
The empirical analysis in the current study is based on data collected by a questionnaire survey conducted in the period 2017-2018 among representatives of 184 project-oriented organizations that operate in Bulgaria.Since there is no specific register or other kind of statistical frame to facilitate a random drawing, the respondents have been selected by a purposive sampling scheme.A specific questionnaire has been developed and sent to 200 respondents (project management experts, project managers, project portfolio managers, and representatives of the top management boards).The method of individual self-interview has been applied by participation in online survey or by submitting a filled questionnaire by email.Appropriate respondents have been reached through the channels of professional networks (LinkedIn; Bulgarian Association for Project Management).All respondents have professional duties and competences in the area of project management performed in a multi-project environment.Moreover, some of them have a key role in the management of a project portfolio operated by the respective organization.Due to substantial non-response 16 questionnaires are excluded from data processing and analysis.

Survey Instrument
For the goals of the current study a two-perspective model of PPM performance has been developed based on a selection of core measures distinguished by short and long-term prospects.The short-term perspective is based on a popular construct suggested by Gardiner and Stewart (2000) that requires evaluation of the three components: cost, schedule, and quality of delivery.It actually evaluates the "cumulative success" of the projects in the portfolio by relating the "average project success" to these components of project management triangle.This approach evaluates the project performance at the level of the entire portfolio by taking into account not only the individual projects' characteristics but also the interdependences between them -this way incorporating the synergy effect.
The long-term prospect is related to the strategic alignment of all projects in the portfolio regarding the organizational strategy.This perspective implies measurement of two main aspects: strategic fit of project objectives and strategic relevance of resource allocation (Hendriks et al., 1999).This way the current study adopts a set of measures regarding the strategic fit reflecting the degree to which: (i) the individual project objectives contribute to the achievement of overall strategic objectives of the organization; (ii) the resource allocation by projects is as much effective as possible in order to provide support to those projects which have highest strategic relevance.
Hereafter a selection of empirical results is presented regarding the evaluation of project portfolio management performance and success in Bulgarian project-oriented organizations.
It is generally of explorative nature and emphasizes on the operationalization of PPM performance and success constructs through appropriate survey tools.The definition of measures and variables is based on conceptual arguments from the specialized literature and suggested solutions from recent literature sources.Among other issues, the questionnaire survey was focused on aspects related to the characterization of the two dimensions of the model (short-and long-term perspective).
The questions used for constructing necessary variables were close-end utilizing a 7-rank Likert scale ranging from 1-"strongly disagree" to 7-"strongly agree".Each item required an opinion concerning a specific aspect of project portfolio success.
Short-term perspective -a set of items capturing the "average project success" by encompassing the criteria of cost, time, and quality (Jonas et al., 2012): • "On average, our projects achieve a high schedule adherence" -schedule accomplishment; • "On average, our projects achieve a high budget adherence" -cost constraints; • "On average, our projects achieve a high quality adherence" -quality attainment.
Long-term perspective -a set of items capturing the strategic fit concerning goals alignment and resource allocation (Biedenbach & Müller, 2012): • "Our project portfolio is consistently aligned with the future of the company"; • "The corporate strategy is being implemented ideally through our project portfolio"; • "We have the right number of projects for the resources available"; • "The allocation of resources (people, time, and fund) to the projects reflects our strategic objectives".
Each set of items has been evaluated in respect of their reliability using Cronbach's alpha and the respective variables for short-and long-term perspectives of project portfolio success have been extracted by averaging the ranks from the individual answers.

Profile of Respondents
A variety of personal attributes, both demographic and professional, were recorded during the survey.The pool of respondents is relatively balanced by gender, however, the age structure shows predominantly middle-aged individuals (over 70% in the range 31-50) and about one fifth with age up to 30.The sectoral structure of employment shows that almost half of them work in organizations operating in sector "IT and communications".The next more frequently recorded sectors are "Public administration" (22%) and "Construction" (14%).
An important characteristic is the experience of the interviewed -over one third indicated a long period of general work experience (over twenty years) whereas about 11% declared just a recent experience: up to 5 years.The professional experience in project management has been identified by the number of years working in project management (project team member, project office expert, project manager, project portfolio manager).The major share (about 60%) is held by respondents with specific experience 6-15 years, and over 20% indicate professional experience over 16 years (Table 1).Additional to the experience, the survey provides information on the positions held at the moment of filling the questionnaire.About one third of respondents act as project managers at their organization -a position which holds the largest share in the sample (Figure 1).About a quarter of the interviewed occupy various positions of project experts, followed by members of project teams (21%) and project office experts (9%).The position of "project portfolio manager" is rarely met, but still, about 8% of respondents indicate such occupation.Albeit rarely, representatives of top management of project-oriented organizations have also participated in the survey.

Descriptive Statistics of the Variables
The results from the questionnaire survey presented hereafter include the descriptive statistics for the selected items and construct variables, the reliability measures as well as the correlation coefficient between the aggregated variables.Table 2 presents the descriptive statistics of all items and derived variables in order to provide an overview of the primary data characteristics.In general, according to these results the mean levels are centered somewhat higher than the middle of the Likert scale, so the answers can be considered as showing higher perception on the status of the items involved.The high average scores by some items (above 5.0) are indicative about the relatively high ranks put by the respondents about the evaluated measures, especially for short-term perspective items.The lowest mean ranks level is observed for the item "The corporate strategy is being implemented ideally through our project portfolio" (4.25) -considering the moderate degree of variation (CV = 19.5%) here we find a relatively unfavorable opinion indicating some degree of misalignment of individual project goals and overall organizational strategy.On the contrary, the highest level is estimated for the short-term item "On average, our projects achieve a high budget adherence" (5.45) which shows a relatively high degree of financial discipline and reliable cost control during the project execution phase (according to the opinion of the respondents).On average, the mean score for the strategic alignment variable (4.92) is lower than the mean score for the average project success variable (5.34).Most likely, this result is due to relatively lower ranks assigned by respondents to items "The corporate strategy is being implemented ideally through our project portfolio" (4.25) and "The allocation of resources (people, time, and fund) to the projects reflects our strategic objectives" (4.36)still, the long-term dimension of portfolio decision making is to gain a better focus by project portfolio managers.

Reliability Measures for Items and Variables
Table 3 presents the results on the internal consistency (reliability) measures obtained by the Cronbach's alpha indicator.Both construct variables have scores around the threshold (0.7) considered as acceptable in respect of the reliability of the instruments.
Although a diversity of practices and related attitudes of respondents is observed regarding PPM processes -which are still in a phase of development in the country -the instrument has proved to be reliable.This is valid especially for the short-term perspective items where the construct variable shows quite high value (0.882); the removal of any of its component items would lead to unfavorable reduction in its value.The situation is not the same with the long-term perspective items, even though the overall reliability of the respective package is acceptable (0.734).Here the result for item 3 "We have the right number of projects for the resources available" shows that, if removed, the reliability of the instrument measuring the long-term perspective (strategic alignment) will clearly increase (0.773).The corporate strategy is being implemented ideally through our project portfolio 0.675 We have the right number of projects for the resources available 0.773 The allocation of resources (people, time, and fund) to the projects reflects our strategic objectives 0.542 In respect of the interrelation between the two perspectives of PPM performance, the Pearson product-moment correlation coefficient has been estimated.Its value (0.784) shows a relatively high degree of covariation of the two construct variables which additionally supports the initial findings and assumptions about the instrument implemented for the goals of this study.There could be a variety of arguments and motivations in search of explanation for the existing discrepancies in the patterns of variation of the two variables and their constituent items, however, these issues are beyond the scope of this paper.

Conclusions
This study provides certain implications about the initial stage of project portfolio management research in Bulgaria.The emerging practices of introduction and enhancement of PPM approach in Bulgarian project-oriented organizations require further investigation of various aspects of its performance, effectiveness, and overall success.This study is focused on the assessment of PPM performance in two dimensions, namely, short-and long-term perspectives, along with their empirical measurement and testing for the reliability of the applied instrument.
The results presented in this study provide orientation about the directions of next stage advancement in the analysis of PPM performance.For instance, a conceptual model of the determinants of PPM performance is necessary to reveal the causal links between the characteristics of PPM practices in project-oriented organizations.Further studies of PPM performance are expected to reflect the degree of maturity of such practices.A further study of the determinants of success and PPM performance is still a challenge for project management research in Bulgaria which can be combined with benchmark evaluations and comparative analyses in search of empirical evidence about a variety of relevant research hypotheses.

Figure 1 .
Figure 1.Distribution of respondents by position held at the organization.

Table 1 .
General and professional experience of respondents.

Table 2 .
Descriptive statistics of items and variables.

Table 3 .
Reliability measures for items and variables.