Performance evaluation of project management system based on combination of EFQM and QFD

Article history: Received: January 1


Introduction
Project management is the application of knowledge, skills, tools, and techniques implemented for the project activities to meet the project requirements.In other words, project management includes activities for planning, organizing, performance supervision and performance guidance, and tries to deliver specified results by proper utilization of resources and with previously agreed upon cost in due time (PMI, 2013).
Nowadays, organizations are facing with various challenges for evaluating their performance.In general, performance means efficiency in addition to flexibility, creativity and continuous improvement.Weaknesses of traditional performance evaluation systems have motivated researchers to find improved approaches to assess the system performance.Therefore, many processes were made to be used in diverse organizations.Table 1 shows a number of evaluation systems (Taticchi et al., 2010).

Table 1
Types of evaluation models and their publication year Period Model (System) Authors 1988 Strategic Measurement Analysis and Reporting Technique (SMART) Cross & Lynch 1989 Supportive Performance Measures (SPA) Keegan et al. 1990 Performance Measurement Questionnaire (PMQ) Dixon et al. 1992 Balanced Scorecard (BSC) Kaplan & Norton 1995 Return on Quality Approach (ROQ) Rust et al. 1996 Cambridge Performance Measurement Framework (CPMF) PMS can be evaluated by some general methods.The PMS, as a whole system, needs to be evaluated at several parts.The evaluation system can be divided into three general categories.They are discussed in the Table 2.The most important part of PMS are stakeholders who can significantly influence the project and organizational performance.They are people or organizations practically involved in project system and can have positive or negative effects on project implementation process or accomplishment of the project.Thus, the project management group must identify the stakeholders at the beginning or even during the project, and evaluate their needs and expectations to insure the project success by managing and supervising these needs and expectations.However, categorize the project stakeholders and the factors affecting them are often hard and project management group barely obtain a complete list of project stakeholders.Based on our knowledge, this paper is the first attempt to achieve critical success factors from stakeholder's vision on PMS.Furthermore, we try to present suitable strategies for PMS improvement based on the critical success factors.Previous studies have emphasized on project management which was defined as a point where three goals: time, cost and unique features meet (Atkinson, 1999;Cooke-Davies, 1990;Pinto & Slevin, 1988).During this period, PMS system was not based on distinctive groups such as stakeholders and most researches were carried out on project management performance evaluation and its multiple effects on project management accomplishment.Kerzner (1987) offered a list of critical success factors in PMS and analyzed the effects of these factors on the stakeholders.Pinto and Slevin (1988) clarified comprehensive list of success factors.The definition of stakeholders during this period was unclear and vague.Morris et al (1987) showed that all stakeholders should be identified through evaluation phase in order to evaluate PMS success.The first research achievement in the relationship between project management and quality management was associated with the effect of project management on the successful implementation of total quality management (TQM).Hides (2000) showed that the effective implementation of TQM needs to have accurate PMS.
To end with, Davis (2014) investigated the important role of all stakeholders in PMS success.In order to evaluate PMS performance one needs to accomplish a comprehensive framework.ISO, Canadian award for quality and reliability, and National award model for organization excellence and EFQM are different methods to define a frame work.Hillman (1994) introduced National award model for organization excellence and efficiency as the most common method for evaluation of organization improvements.A national award model for organization excellence and efficiency was known as a method in definition of total quality management (European Foundation for Quality Management, 1999).QFD is a prevalent quality technique that is settled design quality to achieve well customer expectations (Sullivan, 1986;Akao, 1990).In QFD procedure, a matrix named the house of quality (HOQ) is used to show the relationship between the CRs and DRs (Hauser & Clausing, 1988).Yousefie et al. (2011) used EFQM and QFD to select effective management tools.

Project management evaluation model
The EFQM was established based on a series of fundamental concepts and principles which are necessary for the whole organizational performance (irrespective of organizational size and functions).These concepts consist of result orientation, customer orientation, leadership, stability of aims, reality and process based management, staff participation and development and continuous learning, innovation and improvement, development of partnership and social responsibilities (Eskildsen, 1998;Ghobadian & Woo, 1996;Westlund, 2001;Wiele et al., 2000).Consequently, EFQM is a suitable model for project management performance evaluation.Bryde (2003) presents six the following criteria as Fig. 1.

Results
European Foundation Quality Leadership People

Processes
People Results

Policy & strategy
Customer Results

Quality Function Deployment
QFD is structured approach to define customer requirements and translate them into design requirements to produce products with specific needs.The "voice of the customer" describes these specified and unspecified customer requirements.QFD inspects the market and classify customers and attempts to identify customer's needs through analysis of different processes and considers them in all design and production phases.House quality matrix contains useful materials.The rows indicate CRs and the columns show DRs.The cell where each CR and DR meet is specified by 0, 1, 3 or 9.As a final point, critical factors from the stakeholders' point of view can be found.

Research methodology and Assumption
Our suggested methodology can be described in the following five steps: Step1: Collect PMS elements and evaluate their performances from stakeholders' point of view.
Step2: Preparation, distribution and statistical analysis of a questionnaire based on the European Foundation for Quality Management excellence model.Each of the nine criteria is evaluated through some questions.
Step 3: Determine the project stakeholders' needs and convert these needs to HOQ matrix.Moreover complete critical success factors based on the stakeholder's standpoint.
Step 4: Define a list of critical success factors according to the organization charts.
Step 5: Determine the effects of success factors in selection of R&D projects.

Critical success factors
There are three main steps to achieve CR: Step 1-categorize the responders and determine the importance of each one.
Step 2-collect the questionnaires and determine their importance.To find the importance of customer demands, initially, the importance of customers' demands by consideration of stakeholders without their correlation are calculated, and then importance of customer demands by consideration of stakeholders are normalized according to Eq. ( 2), (Maleki et al., 2010).
Eq. ( 1) shows the importance of ith factor by jth customer when other factors effect on it.The first term calculates the responder i's importance times to the score given to factor jth.The second term shows the total effects of other factors on ith factor, and Eq. ( 2) normalizes this value.The above equations yield the matrix of stakeholders' importance.Table 3 shows mathematical symbols definition.When normalized scores per stakeholders' are calculated, twenty demands with the maximum weights are selected and considered as CR's for the first house of quality.
Step 3-Determine the target values for each DR of the matrix and develope a model for maximizing the stakeholders' satisfaction by considering positive or negative correlation between their needs.
It is clear that increase or decrease in one of the DRs will change the other DRs.Thus, the value of these correlations are specified by signs.In some cases, the specifications have an inverse relationship which means if one of them is considered at a high level, the other one will be ignored.In such cases, making a sort of balance or tradeoff between the specifications is easy.In addition, if both specifications are accessible at high level, indeed, the customer satisfaction is high.Finally, determine importance of qualitative characteristics in order to identify the most important of them.The following steps are suggested to compute the importance of each qualitative characteristic: -Determine the degree of relationship between customer's demands and operational characteristics -Determine the degree of correlation between operational characteristics -Normalize the values of relationship degrees: to normalize values, in case of a correlation between the characteristics, use the following terms (Maleki et al., 2010) .

Importance-performance matrix
Importance-Performance Analysis (IPA) was presented by Martilla and James (1977).This method helps managers achieve a correct understanding of products and services features and focuses on the features having the extreme effects on customer satisfaction to measure the organization performance for those features.The obtained data from customer assessments will be used to make a matrix in which performance is assigned to the X axis and importance is assigned to the Y axis.In this method we have four quarters: First quarter (major strength): If a feature or criterion is assigned to the first quarter of IPA matrix, it means they evaluate the importance of this criterion as good or suitable.
Second quarter (major weakness): If a feature or criterion is assigned to the Second quarter of IPA matrix, it means they have high importance from customers' point of view, but low performance of the organization.
Third quarter (minor weakness): If a feature or criterion is assigned to the third quarter of IPA matrix, it means low importance from customers' point of view and weak performance of the organization.Fourth quarter (minor strength): If a feature or criterion is assigned to the third quarter of IPA matrix, it means low importance from customers' point of view, and high performance of the organization.

Questionnaire analysis
In this paper due to lack of sufficient information and regarding the usable results, the importance value of responders was supposed to be 1.The questionnaire contains 60 questions.It was distributed between thirty internal stakeholders.They were public sector managers, administrative managers, strategic managers and members of project groups.The questionnaire has been designed based on principles of European Foundation for Quality Management (EFQM) and the responders have answered to two kinds of questions: 1-Importance of the criterion in the organization, 2-The present status of the criterion in the organization.
The score has been given from 0 to 10.To conclude, the achieved answers are evaluated by quality matrices.

Statistical analyses
Table 5 shows all the CRs and the scores and Fig. 2 shows the status of each factor schematically.As shown in Fig. 2, when the points get closer to a vertical axis, it needs more attention to the factors that causes dissatisfaction.In other words, all stakeholder's demands whose scores are less than 5 and they are in quarter 2 are the main priorities of organization improvement.Next, these results can be obtained from calculations.The degree to which consultant information is used for project management and presenting new ideas.

7.10
The degree to which consultant information is used for project management and presenting new ideas.

Role of partners in project management
Status of two-way dialog with all stakeholders of the Institute.4.66 6.72 The degree to which external stakeholders affect PMS of the Institute.
5.34 6.69The degree to which internal stakeholders affect PMS of the Institute.
5.38 6.93 The degree to which financial supporters affect PMS of the Institute.
5.93 7.76 The degree to which internal stakeholders are satisfied with PMS of the Institute.
4.90 7.45 The degree to which external stakeholders are satisfied with PMS of the Institute.5.76 7.24

Using quality function deployment (QFD) for evaluations
The points in the second quarter of Fig. 2 which are extracted from Table 5 (customer demands) can be considered as an equivalent of CR.As a result, in order to satisfy these demands, organization strategies need to be defined.In table 6, twenty important strategies have been explained.0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 Paying attention to all key factors of project management (time, cost, scope and quality) 4 Improving the general knowledge 5 Information system development 6 Increasing the manpower interest, perseverance and motivation 7 Increasing the learning spirit in the manpower 8 Creating a two-way dialog between the staff and organization 9 Planning for improving the human resources 10 Increasing the staff partnership in the organization issues 11 Incorporating information and knowledge management 12 Paying attention to key processes 13 Gratifying the staff 14 Financial resource management 15 Leaders interaction with customers and stakeholders 16 Outsourcing management 17 Identifying knowledge of the staff 18 Creating a culture for continuous improvement 19 Reducing the financial risk 20 Technology and method development House of quality has three sub-matrixes which are defined as below: a-A matrix dealing with the relation between customer's demands and their effects on each other.b-A matrix dealing with relation between strategies (DRs) and their effects on each other.c-A matrix dealing with the relation between customer demands and defined strategies.Quality function deployment matrix deals with analysis of dependency and effects of customer needs (CRs) as well as dependency and effects of technical needs (DRs).This dependency can be between -3 and +3.The numbers inside the matrix are 0, 1, 3 and 9 based on a scoring system, and after entering the relations, they are as in the annex.Table 7 is the last step of this research and shows the strategy prioritization after analyzing the demands and completing the House of quality matrix.Followed by IPA figure all stakeholders' demands are explained separately for EFQM factors.As shown, all points in second region can be defined as critical factors.In fact, these points express demands which are very important and currently they are defined as weak points in the organization.Also, the points in the first quarter are strong points of the organization which should be kept.Fig. 2 clearly shows that the factors associated with leadership, which are explained in Table 5, are often considered as weak points.

Discussion
In order to discover the critical factors, all parts of project management investigate through a comprehensive questionnaire.In performance measuring phase, organizations pay much attention to their financial indexes, while financial issue is the only one part of organization performance which is associated with the other factors performances.Moreover, financial indexes mostly indicate the organization performance in the past.
EFQM is formed by two fundamental factors which are "Enablers" and "Results".Terms and concepts incorporated in this model are so general that they can be interpreted in a different way, and organizations can make diverse evaluation indexes with these terms.Quality function deployment can solve this flaw.The main goal of project management is to keep customers and stakeholders fulfilled, and the main objective of quality is to satisfy customer needs.Therefore, to succeed in keeping different stakeholders satisfied, we should use the quality management concepts.
Various methods can be used to design the questionnaire.In this research, excellence quality management principles is considered as a basis and foundation for making the questionnaire.Table 8 shows the normalized importance average.The numerical value 4.744 of 10 indicates the present status of an organization in the PMS with regard to 7 factors of EFQM.Increasing this value means organization status in the PMS is gaining to improve.It should be noted that each stakeholder demand importance value is based on the demand's effect which is obtained via relation 1.Finally, twenty important strategies, which are obtained from quality matrix, are ranked in Table 7 and can be implemented as PMS improvement strategies in the respective organization.

Conclusion
This research has attempted to use the combination method of European Foundation for Quality Management excellence model and quality function deployment to discover present system critical success factors in the organization, and develops system improvement strategies to select projects.
According to Evaluating of system, the weak point of PMS is PMS staff, and the strong point of PMS is the effects of factors on one another.This research has shown that the effects of different indexes of EFQM on one another must be considered to discover the weakness and strength of PMS of the organization.Also, implementation of every strategy has simultaneous positive or negative effects on the demands.These effects can be considered by considering QFD principles.
The study raises several issues that could implement in further research.Below, some of these extensions are presented: 1-Fuzzy QFD is one of the important methods in project management discussions.Thus, fuzzy principles can be used to develop project management.2-Implementation of desired strategies is another development method.Each strategy requires a definite executive method, and then these strategies must be evaluated and controlled.3-Each responder has a special importance in an organization.Since there was not sufficient information available in this research to obtain the weights, all the weights are considered to be the same.These weights can be assumed to be different in future researches.4-Balanced score card principles can be an option to develop this method rather than considering EFQM as another evaluation index.5-Creating a mathematical model by using data covering analysis to evaluate PMS can bring researchers more tangible results.

Table 2
Different methods of PMS evaluation

Table 3
Concept of mathematical symbols

Table 4
Concept of mathematical symbols

Table 5
The results obtained for all CRs based on the questionnaire

Table 6
Strategies of the organization (DR)

Table 7
Strategies obtained from quality matrix

Table 8
Organization status investigation after using quality matrix