Project management frameworks and practitioner preferences for capturing lessons learned on projects

This study focuses on how practitioners view learning activities in projects and how these activities are influenced by the approach or emphasis espoused in different reference texts. The perceived importance of learning lessons in projects was surveyed in the literature, establishing the difficulties encountered, various methods for process improvement and the realities of current practice. The practices of individuals working within two major project management frameworks used in developed English-speaking economies were compared. A survey was constructed to address the question. Data from a pilot survey support the theory that the PMBoK and PRINCE2 are not perceived as synonymous and that differences will be more evident with a larger dataset. Inferences are drawn linking the pilot survey to the outcomes of previous studies and making recommendations for further research.


Introduction
opens his article on identifying the hard lessons from projects with the observation that "although it is generally accepted that learning lessons from projects is important, a number of authors have noted the lack of project post-mortems occurring in practice".It has also been noted that while many project management practitioners agree that capturing lessons learned from projects is important, not all of them actually do so (Keegan & Turner 2001;Kotnour & Vergopia 2005).
This study focuses on how practitioners view learning activities in projects and how these are influenced by the approach or emphasis espoused in different reference texts.The practices of individuals working within two major project management frameworks used in developed English-speaking economies will be compared.These are: • Managing Successful Projects with PRINCE2 (Office of Government Commerce 2002), which purports to provide a complete methodology including document templates for all project management processes.• The Guide to the Project Management Body of Knowledge (PMBoK, Project Management Institute 2008), an official standard in the United States (ANSI/PMI 99-001-2008), which purports to provide a set of tools and techniques that can be applied to managing projects.
PRINCE2 was written for and by the United Kingdom government and has been widely adopted by government agencies in many Commonwealth countries.Government approaches to organisation and risk tend to be structured and conservative.It is an approach that suits some circumstances but not all.
I was intrigued by the possibility that project management practitioners agree on what the best practice in the area of capturing lessons learned in projects (capturing knowledge for future use) is but are choosing not to apply it.This was coupled with an interest in whether the level of individual judgement they can exercise within their preferred project management framework affects such decisions.
This study aims to establish the significance of the preferred project management framework on a project manager's decision regarding whether or not to capture lessons learned from projects using the tools espoused in the frameworks.

The importance of learning lessons in projects
It seems generally accepted that learning lessons in projects is important.Cooke-Davies (2002: 189) for instance identifies "an effective means of learning from experience" as one of 12 "real" success factors in projects.The shift to the knowledge-based society (Kotnour & Vergopia 2005) means that competitive advantage now resides in the skills of an organisation's human resources (Eskerod & Skriver 2007) and having experienced project managers is one of the keys to project success (Petter & Vaishnavi 2007).
On the flip side, there is also much discussion of the failure to learn from project experience (Robertson & Williams 2006) particularly in the field of software development (Petter & Vaishnavi 2007) and in complex projects (Lyneis, Cooper & Els 2001).

Difficulties encountered
Many difficulties have been identified with the process of learning in projects.Williams (2003: 443) notes that "we have yet to discern how to systematically extract and disseminate management lessons as we move from project to project".Individual perspectives can also hinder learning in projects, such as when learning is seen as being separate from official project activity (Kotnour & Vergopia 2005), or has not proved helpful in the past (Williams 2003).
Individuals may also experience "difficulty identifying 'hard', non-intuitive lessons from projects such as those resulting from feedback and dynamic, systemic effects" (Williams 2004: 273).They may suffer from hindsight bias, which can distort the interpretation of past decisions and increase the blame that is apportioned to participants (Bukszar & Connolly 1998).They may also have difficulty observing their own responses to events (Williams 2003) or may not want to acknowledge evidence of personal failures (Robertson & Williams 2006).
Organisational constraints can also hinder the learning of lessons in projects.Cooper et al. (2002) note that little or no time may be given to project reviews, especially when the project has been a failure.Williams (2004) describes the approach of BP's Post Project Appraisal unit, which focuses on "what happened" rather than "why".This leaves a key aspect of project learning unexplored and may be a contributing factor to why lessons learned activities in projects have not been seen as good at generating new insights (Robertson & Williams 2006) or operationalizing tacit knowledge (Williams 2008).

Methods for improvement
A number of approaches have been proposed to improve matters.Many note that both the PMBoK and PRINCE2 advise keeping a database or log of lessons learned.This seems to be by far the most common practice but is also widely seen as ineffective (Keegan & Turner 2001;Newell 2004).Whereas PRINCE2 embeds its lessons learned processes into a methodology, the PMBoK leaves the use of any tool or technique to the judgement of the project manager.This lack of obvious effectiveness could be the source of differences in application between students of these two frameworks.Keegan and Turner (2001), Williams (2003) and Newell (2004) all refer to Nonaka and Takeuchi's work on the knowledge creating company and that knowledge is much more than information.Several authors have proposed approaches that engender learning by promoting reflection on past events: the Plan-Do-Study-Act cycle (Kotnour 1999); Cognitive Mapping (Robertson & Williams 2006); and full System Dynamics modelling (Williams 2003(Williams , 2004;;Lyneis, Cooper & Els 2001;Cooper, Lyneis & Bryant 2002).

Current practice
So what are practitioners actually doing?Authors vary in their opinions.Williams (2003: 445) notes that "in most individual companies in practice, project review processes are rarely in place".Whereas Newell (2004) notes that most (IT) organisations had instituted project reviews but had not found them to be very effective.Kotnour and Vergopia (2005) briefly mention an empirical survey of 27 multinational companies, four out of five of which did not conduct post-project reviews.Keegan and Turner (2001) describe a detailed study of 19 organisations, all of which have knowledge-retention processes that are rarely used.Kotnour (2000) surveyed 43 project managers at a local Project Management Institute (PMI) chapter meeting.Thirty-one (72%) did conduct project reviews with two major drivers; an organisational directive and a desire to make improvements.The 12 who did not cited lack of time, uniqueness of the projects and no organisational directives as the main reasons.Williams (2008) surveyed 522 senior project managers from the UK, US and China.Ninetysix per cent were members of the PMI, 45% held the Project Management Professional qualification and 9% some other project management qualification.Sixty-two per cent responded that their organisation had lessons learned procedures but only 11.7% said they were closely adhered to.
The consensus seems to be that most firms fail to review projects and as a result tend to repeat errors and make bad decisions (Robertson & Williams 2006).Alternatively many organisations have processes in place but are finding them ineffective.Eskerod and Skriver (2007: 113) note in one case study that "the fact that the initiatives did not contribute very much to knowledge transfer was quite a puzzling finding" and proceeded to re-evaluate their data from new perspectives.
It seems that project managers are choosing not to engage in these activities when they have the freedom to do so.This study seeks to compare the prevalence of lessons learned activities within the PMBoK and PRINCE2 project management frameworks, where freedom of choice is assumed to be greater under the PMBoK.

Methodology
This study begins with the question: Does a project manager's preferred project management framework affect the prevalence of lessons learned activities?The hypothesis being that the choice of project management framework does affect the conduct of lessons learned activities, with adopters of the PMBoK conducting fewer activities that all see as relatively ineffective.
Having begun with a hypothesis and seeking to investigate attitudes and opinions, a survey approach was chosen for the methodology.The survey focused on identifying differences between adopters of the PMBoK and PRINCE2 on the assumption that individuals and organisations within the developed English-speaking world tend to choose one or the other as the basis for their preferred project management approach.
The survey was designed in five sections based on factors identified in the literature as being associated with learning within the project environment.
Section 1 focused on establishing the presence and use of guidelines at the respondent's current workplace and the actual prevalence of lessons learned activities.
Section 2 focused on establishing the strength of the respondent's personal commitment to a particular framework and their attitudes towards frameworks in general (i.e. are respondent's following a framework because they choose to).
Section 3 focused on establishing the strength of organisational preference for a particular framework, the organisation's overall management style (formal/laissez-faire) and level of project management autonomy for deciding project management processes and procedures (i.e. are respondent's following a framework because they are told to?).This last factor is useful in establishing the organisation's approach to knowledge management and whether it attempts to codify experience or allow it to remain personalised in its human resources.
Section 4 focused on a number of concepts identified as important from the literature review.Three dimensions of organisational culture were measured: forgiving/blaming; conservative/risk taking; and supporting/demanding. Organisational structure was identified, whether functional, matrix or project based.The coverage of knowledge management in the organisation was measured, and whether tacit knowledge was included or just explicit knowledge.Finally how well the organisation was able to define both project goals and methods was measured to enable plotting of the organisation's activities within Turner and Cochrane's matrix of project types (Turner & Cochrane 1993) Section 5 focused on the independent variables of age, sex, qualifications, years of experience etc, for the respondent, plus industry type, number of employees etc, for the organisation.
For sections 1-4 respondents were asked to describe, using a seven-point Likert scale, the degree to which their opinions and practices aligned with a series of statements (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree, 6 = not applicable, 7 = don't know).The statements were designed in pairs, threes or fours, with one statement framed in the negative to reduce the impact of respondents providing response sets.
The statements for sections 1-4 were then mixed and renumbered so that the survey presented to respondents included one group of 37 statements requiring Likert scale responses and a second group of 11 questions requiring specific data.
The survey was constructed as a Microsoft Excel file with two worksheets.The first worksheet contained a covering letter describing the background to the research and a short description of the target population, i.e. those who are aware of the PMBoK and/or PRINCE2 and who have managed a project.
The second worksheet contained the survey questions.Excel was chosen to allow electronic distribution and retrieval, to enable fast data entry and coding of responses and because the worksheets could be formatted to also produce a printed version for manual completion.
The survey was piloted with a convenience sample of seven attendees at an evening class on a Master of Project Management course at the University of Technology, Sydney and modifications made based on their feedback.

Findings
The pilot group have some weaknesses as a sample population.The group averages 3.2 years of experience managing projects.However, the mode is only 1 year and the median 2 years, showing that the data in this small sample are significantly skewed.
The group also lack any adopters of PRINCE2 as can be seen from Figure 1, which shows a scatter diagram of the two questions regarding personal opinions on the PMBoK and PRINCE2.

Figure 1. Pilot population lacks PRINCE2 adopters
This makes it impossible to address the core question by comparing adopters of the two frameworks.However, there are a number of conclusions that can be drawn from even this small sample that point to how the analysis of a larger sample might be approached, given that the data remain representative of the target population.
Figure 2 shows a scatter diagram of responses to questions on the presence of formal guidelines for capturing lessons learned and general adherence to guidelines.It shows that while the majority of respondents' organisations have guidelines they are almost universally not adhered to.

Figure 2. Guidelines rarely adhered to
This is the same finding as from Williams' (2008: 257) larger survey and indicates that the data may indeed be representative.
The pilot group also represent an interesting mix when plotted on Turner and Cochrane's goals and methods matrix, Figure 3 below, with the majority responding that methods are not well defined within their organisation.
Formal procedures and/or guidelines are generally not adhered to in my organisation.
My organisation has formal procedures and/or guidelines for learning lessons from projects.

Figure 3. Goals and Methods matrix
This contrasts with the result in Figure 2, where the majority responded that guidelines were available.The guidelines perhaps do not amount to methods for action.
And, again, when the presence of guidelines is plotted against the capturing of lessons learned on all projects, Figure 4 below, a very strong relationship is evident.Without guidelines no one captures lessons and with guidelines no one says they do not capture lessons.
This also seems difficult to reconcile with the result in Figure 2 where guidelines are rarely adhered to and yet when they do exist they seem to be ensuring that lessons are captured from projects.
In my organisation the methods for achieving project goals are well defined.
In my organisation project goals are well defined.

Figure 4. Capturing lessons on projects
Perhaps the questionnaire should be redesigned to develop more precision around the concepts of guidelines, methods and capturing lessons to better establish what is actually going on.
Pearson's r was used as another method of two-factor analysis to look for linear relationships amongst the pilot data as a guide to further exploration, see Figure 5 below.The strongest relationships, those with r values of 0.75 and above or -0.75 and below, were mapped to look for patterns of association, see Figure 6 below.
In Figure 6, for instance, Age relates positively with Years of Experience and Salary, which would be expected and indicates that other relationships may be reliable and not artefacts of the small sample size.Bearing in mind the low confidence levels possible with the available data a number of inferences might be drawn from the pilot data for discussion.
"Lessons learned" are captured from all projects.

Guidelines & LL on All projects
Have you ever held a 5.7s  Of the 48 datasets in the survey mapped onto Figure 6, only four are without high magnitude relationships with other datasets (using Pearson's r) these are the uncoloured boxes at the bottom of Figure 6.The remainder of the data form two clusters.The left-hand side of Figure 6 (blue boxes with square corners) forms one cluster and the right-hand side (green boxes with rounded corners) the other.The data from the statement "There is an active Project Management community within my organisation" (yellow oval) forms the only link between the two clusters.
This result indicates that there may be significant underlying structure to the data and that Principle Component Analysis (on a larger dataset) would prove useful in establishing what new factors or variables better describe the variation observed.Analysis of Variance (ANOVA) could then be used to establish more rigorously the relationships between the various factors or variables.
The dataset with the highest number of strong positive relationships is " 'Lessons learned' are captured from all projects".This links to having formal guidelines and procedures, having a tailored project management approach, having an active project management community, seeing lessons captured by other teams as useful and the size of projects conducted by the organisation.The dataset with the next highest number of strong positive relationships is "Number of employees".This links to standard procedures being followed, having an active project management community, years of experience and salary.
The dataset with the highest number of strong negative relationships is "It is difficult to identify profound lessons from every project".This links to standard procedures not being followed, not having an active project management community, not having a documented methodology, not seeing accredited training as important and number of employees (i.e.fewer employees makes it more difficult).The dataset with the next highest number of strong negative relationships is again "'Lessons learned' are captured from all projects".This links to seeing professional certification as important, having an unforgiving organisational environment and actively capturing lessons, not only from big projects.
This pictures the pilot group's opinions as being in agreement with the literature where the successful capturing of lessons from projects is dependent on a number of factors.There need to be formal guidelines that are tailored to the circumstances and then followed.There needs to be an active project management community with a culture of learning and at least some experienced members.The organisation needs to be sufficiently large to develop these attributes and it must hold project managers to account for project outcomes.

Discussion and conclusion
The data obtained is insufficient to either accept or reject the hypothesis that a project manager's preferred project management framework affects the prevalence of lessons learned activities.However, the design of the survey is adequate to address the question with some minor design modifications and a larger dataset.
From Figure 6 it seems that the two frameworks are not seen as synonymous.The statements regarding whether or not a good project manager needs to have read the framework document or whether organisations prefer to recruit certified project managers link to different issues for the PMBoK and PRINCE2.These differences are worth further exploration.
The centrality of an active project management community in Figure 6 is also worth further exploration.The principle questions being whether the same two-cluster structure reappears with a larger dataset (say >100 active professionals representative of the two frameworks) and whether an approach like Principle Component Analysis can identify the factors central to each cluster.
Project management is still a young discipline compared with many contemporary business practices and the field is not yet defined in terms of dominant schools of thought.Nor is it seen as homogeneous, with only one set of ideas about how to go about managing projects.
The PMBoK and PRINCE2 frameworks though have both been in existence for more than a decade, long enough for them to have had an effect on the way practitioners think about managing projects.
This study has focused on how practitioners view learning activities on projects but there may be other aspects of project management that are influenced by the approach or emphasis espoused in the different reference texts.Project organisation and project risk management might also prove fruitful areas of investigation.
Project management practitioners need to make choices about what to do and what not to do to achieve their goals and, later, about what worked and what didn't work in their particular circumstances.How the different frameworks adopted by practitioners affect this learning process is a question that remains unaddressed by the literature.