Dissertation by portfolio: an alternative to the traditional

one copy of any article(s) in SHURA to facilitate their private study or for non-commercial research. You may not engage in further distribution of the material or use it for any profit-making activities or any commercial gain.


Introduction
This paper will consider dissertations on taught MSc degrees in computing disciplines at Sheffield Hallam University. Traditionally the dissertation is the last module a student studies and is designed to be a showpiece of their work and interests developed while studying the taught component of the course. Depending on the structure of a 180 credit degree, this is normally either a 45 or a 30 credit module. The resultant thesis is approximately 10,000 words (30 credit) to 15,000 words (45 credit).
The Department of Computing is part of the Faculty of Arts, Computing, Engineering and Sciences. The number of M-level (masters level) students in this department who have English as a second language (L2) has increased from 53 in 2004/5 to 203 in 2010/11. This made up 84% of postgraduate computing students in 2010/11 which compares to 36% in the 2004/5 academic year.
One of the hardest things any student has to do as part of their postgraduate studies is to write a traditional thesis. This is not easy for native speakers, but the difficulty is compounded for L2 students. Poor English expression is penalised by marking schemes that assign weight to English usage at Sheffield Hallam University. Not that Sheffield Hallam University is alone in this. For example, Seymour (2005) discusses new approaches to assessment criteria at M-level, but still retains a component mark for English usage.
The traditional dissertation is the cumulating task of an M-level degree, but it is not an end in itself. The dissertation is an instrument (or artefact) used to assess how well a student has met the learning outcomes in a module based around an individual research project. The argument to be put forward in this paper is that a dissertation in academic English is not always the most appropriate way for a student to demonstrate meeting the learning outcomes. An alternative dissertation model composed of a portfolio of related artefacts in a variety of media presenting evidence of attainment will be presented.
The results of a pilot of students preparing a dissertation by portfolio conducted during the summer of 2010 will be discussed. The lesson learned from this are being used to refine the project guidelines for the next cohort of students and their supervisors.
This paper draws on and expands the previously published paper, 'Dissertation by Portfolio -a break from traditional approaches', by Author (2011)

Dissertations
A lot has been written about L2 students producing dissertations. This literature has mainly concentrated on elements such as the importance of English grammar and correcting it. Truscott (2007) claims that grammatical error correction has at best a negligible effect and at worst can be marginally detrimental. His study found that students who had been subjected to grammar correction tended to shorten and simplify their writing to avoid being penalised. Bruton (2009) claims the debate is about the last P in the PPP (Present-Practice-Produce) sequence. In all the literature, the product (dissertation) is seen as written evidence of the Present-Practice part of the sequence. Although the written form may be electronic (Microsoft Word, pdf format or similar) other presentation formats not in academic writing form are not considered. Bitchener and Basturkmen (2006) concentrate on the understanding of the function of the 'discussion of results' section of a thesis. They acknowledge the benefits of a viva voce examination, but don't consider other alternatives to written evidence. Even the viva voce examination does not generally result in a persistent artefact (such as video or audio file) which is unfortunate as a verbatim recording of the discussion could be useful for future research.
The "speech-writing hybrid of computer mediated discussion" is considered by Warshauer (2001) who was investigating the role of 'new technologies' in academic discourse. This provides students with an alternative method to writing of providing evidence of meeting a learning outcome. Warshauer also raises the issue of L1 formalism and its relation to students who are L2.
In the Department of Computing at Sheffield Hallam University, the 'traditional' Mlevel written dissertation of approximately 10,000 words is generally recognised as an opportunity for a learner to demonstrate a variety of M-level characteristics such as:  higher-order problem solving;  the use of analytical skills for complex problems;  the selection of rigorous approaches and the presentation of data, leading to substantiated inferences;  an ability to ground new work in the context of existing, peer-reviewed research;  deep evaluation of both product and process;  written communication.
Viva voce examinations are currently rarely used, but where they are, oral communication, presentation and inter-personal skills are also tested. Arguably any of the above characteristics can be demonstrated in other ways, and in fact the written format of the dissertation may not be the best way for a student to demonstrate them, particularly L2 students as noted by Truscott (2007).
In particular, the final characteristic of 'written communication' may serve to constrain the learner unduly in their ability to demonstrate the other characteristics. This is particularly so for L2 students as noted by Truscott (2007). It should also be noted that students with English as a first language may also have problems, for example students who are dyslexic. Dyslexic L2 students would probably be an extreme case. Related to this is the increased demand from employers for potential employees to demonstrate 'real-world' skills, which may in fact, be masked by the production of a written document. Although employers also want graduates who can communicate in writing most L2 learners from outside the European Union must return to their home countries on completion of their studies because of current visa regulations. It is therefore unlikely they will be writing in English when they gain employment.
Recognising that there has been a radical shift in the Postgraduate learner population in computing at Sheffield Hallam in favour of L2 learners, the appropriateness of a written dissertation for all students needs to be questioned. The ability to write academic English should not be the determining criteria to assess a student's achievement. This is also an opportunity to explore whether the dissertation process can be enhanced to improve the flexibility of contributions that might be demonstrated as being M-level.

Portfolios
A separate theme in the literature from written dissertations and their ability to reflect the performance of L2 students is the use of portfolios in assessment. The majority of the literature here concentrates on undergraduate teaching and assessment. In these cases the portfolio often consisted of unrelated pieces of work, rather than artefacts relating to an integrated project. Barrett and Carney (2005) raised the issue of conflicting paradigms in portfolio development. They identified 3 paradigms; portfolios for learning, portfolios for accountability and portfolios for marketing. The paradigm being considered here is portfolio for accountability, which is to demonstrate an ability to develop an M-level dissertation which will be awarded a grade. This is necessarily in conflict with the second objective of a dissertation which is to facilitate the students learning. A difference is that Barrett and Carney were considering the development of the portfolio over time, but in this case, the time frame is very limited.
Woodward and Nanlohy (2004) saw portfolios as a resource to be used after graduation by students on a teacher education course. The process of creating it allowed students to critically reflect on their learning. This was an example of portfolios for learning.
In contrast, Barrett (2011) considering e-portfolios in a business school context saw them as more integrated and a resource to help graduates gain employment. This is portfolio for marketing. Barrett also saw portfolios being created as a collaborative effort with the academic tutor. This approach does not fit the assessment model (portfolios for achievement) being considered here, but does demonstrate the value of a portfolio on graduation. Klenowski et al (2006) did consider portfolios in postgraduate programmes (but not as an alternative to a dissertation or thesis). They raised the issues of what a portfolio is and the problems of student perceptions. Clarity of purpose was raised as an issue with a tension between gathering evidence (artefacts) and analysis and integration. Analysis and integration is a key to the concept of dissertation by portfolio if the portfolio is to avoid becoming a collection of unconnected artefacts.
Although not explicitly discussing portfolios, Edminster and Moxley (2002) presented the idea of electronic theses and dissertations (ETDs). Although these could be in the form of a pdf, they are 'increasingly in more sophisticated formats such as HTML and XML and include colour images, streaming multimedia, animation and interactive features' (p. 90). Although this suggests different media can be used in an electronic thesis, it is not in itself a portfolio, rather a means of embedding non text media in a dissertation. Pullman (2002) put forward a similar argument and did explicitly mention 'portfolios', but saw the portfolio as a single artefact rather than a collection of artefacts. Barrett (2000), looked at the development of portfolios and her work goes some way to resolving the clarity of process issue. Also working in the context of teachers developing electronic portfolios, she defined the development of a portfolio as consisting of the following stages:  Define the context and goals  The working Portfolio -collect, interject  The reflective portfolio -select, reflect, direct  Connected Portfolio -inspect, correct, connect  Presentation portfolio -celebrate.
In Barrett's terms, these portfolios had the objectives of being a method of professional development resulting in a teaching resource. It therefore was a different context to M-level information technology dissertations, but the development stages provide a useful roadmap to portfolio development in general.
In the case of dissertations, the portfolio is seen as a means of supplementing the traditional dissertation format so as to enable a wider range of relevant skills and characteristics to be presented. There are also potential benefits in terms of how the process of building a portfolio can further improve learners' abilities, especially for employment and life-long learning, a primary concern of Barrett (2011).
Therefore, a portfolio at M-level must be more than just a collection of pieces of work. There needs to be an integrated theme which links the artefacts in the portfolio (and at least one artefact should be used to evidence this). The learner must demonstrate that they are able to produce evidence of their work, but also be able to appraise and select appropriate examples that justify their claim to be at M-level. This corresponds with the views of Kimball (2005) who discusses the central concepts of the pedagogy related to portfolios. Table 1 illustrates the constituent parts of a portfolio. Each of the table elements can be regarded as evidence. This evidence then needs to be tied to the learning outcomes of the dissertation to make sure it is relevant and part of a cohesive whole.

Evidence
It is important to understand what is referred to as evidence. For the purposes of a portfolio a piece of evidence can be one or more artefacts and associated annotation and critical reflection. These three aspects are identified in the grey boxes of Table  1.This evidence should be linked to the learning outcomes of the portfolio module.
It is the understanding of what evidence is that serves as the key differentiator of a dissertation by portfolio from a more traditional dissertation. If a learner chooses to base the portfolio upon their own development as a professional, the learner themselves would be a product upon which a developmental process is applied. In this case there is a possible link between a portfolio for learning and a portfolio for accountability paradigms as defined by Barrett and Carney (2005). A dissertation by portfolio is a format that lends itself to the more flexible style of presentation that might be required to demonstrate some of the creativity that could emerge.

Artefact (many per DbP)
A tangible product that documents an activity or experience. This can be an audio file, video file, picture or pictures, written text (report, email, feedback, online discussion, project plan), diagram, table, model, design or anything that is produced as a result of some relevant activity.

Annotation (many per DbP)
A description that shows what, by whom, when and why the artefact was produced. It should also explain 'what' it shows; some problem solved or learning progression. For instance a learner might select an artefact produced as part of work performed with others. What was the individual learner's (whom) contribution? When was it produced (at the outset as an indicator of their prior understanding; part way through; at the end as a demonstration of development)? Why was the artefact produced?

Critical reflection (many per DbP)
Looking at the artefact and its annotation, critique the inclusion of the artefact and state what has been learned as a result. It would be prudent to indicate how the learner's thinking has led to the production of this artefact, and how the artefact has led to changes in how they will approach future tasks as a consequence. Why was it included within the portfolio? What does it illustrate about their learning?

Overall reflective commentary (one per DbP)
A scholarly piece of writing that connects the evidence together to 'make sense' of the individual items. Essentially this aspect takes the individual pieces of evidence and tells a story about what has been achieved over the duration of the work. The 'scholarly' aspect is satisfied by an individual relating their ideas to grounded literature. The primary emphasis here is about reflecting about the processes used and experienced whilst conducting the work. This may be informed by industrial approaches (software development or project management models) and also developmental approaches (professional development frameworks, learning theories, communities of practice, etc.).

Index/contents (one per DbP)
Some form of indexing that enables the reader to navigate the portfolio. The structure of the portfolio must be made explicit and should provide at least two routes through the content: 1. A defined route that indicates which pieces of evidence should be looked at in which order, and 2. A table of contents that allows an assessment of individual pieces of evidence to be scrutinised in any order.
Option 1 might be provided as a by-product of the 'Overall reflective summary' if the narrative itself clearly indicates which pieces of evidence are specifically referred to. However the inclusion of the reflective summary does not necessarily mean that it has been presented in a way that Option 1 above has been satisfied.

Mapping matrix of criteria to evidence (one per DbP)
A table that relates each piece of evidence to the relevant assessment criteria. If the portfolio is electronic, this can be used as a supplementary means of navigation by hyperlinking references to each piece of evidence. The basis of evidence is the artefact or artefacts that are produced by the learner or as a result of their experiences. Examples of artefacts include:  scholarly/technical/reflective/creative writing  description of an experimental method/approach  critical evaluation of an idea/method/approach  audio recording of a structured interview  video recording of a tutorial or a topic being explained  screencast of an application being demonstrated  excerpt from an online discussion/email exchange  edited collection of singular artefacts into one artefact -a collection of annotated screen shots describing a complex topic/implementation approach/software design, etc.
Of course this is not an exhaustive list. A portfolio would normally consist of many pieces of evidence.

An Integrated Portfolio
The dissertation by portfolio differs from a more general portfolio in that it needs to be integrated to describe a body of work answering a research question. To tie the pieces of evidence together to create an integrated portfolio, there also needs to be:  Overall reflective commentary;  Index or contents;  Mapping matrix of criteria to evidence.
The last two points can be combined. Their primary purpose is to help the reader of the portfolio to navigate the component artefacts.
The subject of the portfolio should be explained in the overall reflective commentary.
In that respect it performs the function of the introduction, introducing the research question and objectives. It should show how the component artefacts generate the evidence required to meet the dissertation modules learning outcomes. Finally it should present the conclusions of the project.

Portfolio Development
The creation of the portfolio would be a negotiated project between the learner and the supervisor. The process of developing a portfolio is shown in figure 1. This roughly follows Barrett's (2000) steps in the development of a portfolio and addresses the issue of clarity of process raised by Klenowski et al (2006).

Assessment criteria
The learning outcomes for the dissertation module are as follows:  Critically review the relevant literature within the domain of discourse and identify, set and justify the focus for the investigation  Select, apply and evaluate a suitable methodological approach  Conduct the research programme and discuss the outcome of the research  Draw valid conclusions from the evidence gathered and produce an academic dissertation at M-level that formulates an argument.
These are reflected in the assessment criteria provided to students in the form of a marking grid:  Knowledge of the domain;  Justification of the approach;  Description of the research and discussion of the outcomes;  Quality of the report (portfolio in this case) and presentation of the argument.
Whilst the learner undertaking a dissertation by portfolio may have engaged in a different learning process to one presenting their work in a traditional written format, the assessment criteria are the same.
For each of the assessment criteria with the exception of the last, a dissertation by portfolio must include at least 3 pieces of evidence, one for each of the criteria and hence learning outcomes. Therefore in total, a dissertation by portfolio will normally present:  A minimum of 3 artefacts each with an annotation and critical evaluation (9 pieces of evidence);  One overall reflective commentary;  An associated index or contents to enable navigation of the content;  A matrix that relates the assessment criteria to each piece of evidence (which may be incorporated in the contents).
The portfolio must include artefacts which cover the learning outcomes of the dissertation module to be complete. The overall reflective commentary and the matrix should be judged within the portfolio as a whole and do not require specific assessment criteria of their own. Having said that, they would be taken into account by the 'quality of the portfolio and presentation of argument' criteria. In particular the reflective commentary will serves to substantiate all four of the dissertation assessment criteria.

Results of the pilot
In the pilot during the summer of 2010, 4 students elected to submit dissertation by portfolio out of a total of 92 computing students. Of these 3 passed, but none gained more than 60%. The remaining student failed to submit. The overall average score was comparable with students submitting a traditional written dissertation, but this sample was too small to correlate with Chang's (2008) study. In this it was suggested there was no significant difference in student achievement on portfolio dissertations compared with traditional dissertations. However it was found that self-perceived learning performance. This study was carried out on a population of high school students studying computing and its applicability to M-level students needs to be tested. The second pilot using documentation and guidelines created during the first is currently underway with 5 out of 79 students opting to do a dissertation by portfolio.
The templates developed in the first pilot to guide students in the development of their portfolios included:  Artefact template  Annotation template  Critical reflection template  Overall reflective commentary template  Contents template As well as these, other guidelines were developed. For example, an important factor (in the students' eyes) was an indication of how a non-text artefact compared to the word limits set for a traditional dissertation. Guidelines for word length equivalences of a range of artefact formats were produced.
At the end of the pilot students were interviewed about their experiences. All students said they found the process harder than they were expecting, however all said they would choose a portfolio approach over a traditional approach if they had to make the choice again. Feedback from the students on the first pilot suggested a sample of a dissertation by portfolio be made available. To facilitate this, an exemplar dissertation by portfolio was developed by the author and put through the assessment process in the normal way. Two academics marked and commented on how the portfolio could be improved. The exemplar was not intended to be a 'perfect example', but rather an example of a pass level piece of work to serve as a basic benchmark for the next cohort of students.
The templates, the sample dissertation, its assessment and feedback and other resources have been made available via Sheffield Hallam University's virtual learning environment to the current (summer 2011) cohort of students.

Future Work
The second pilot is being conducted over the summer of 2011 and students have been given the aids developed in the first pilot. They have also been given access to the sample dissertation by portfolio and assessment comments. The evidence from these new portfolios will be used to further refine the dissertation by portfolio process. Students completing a dissertation by portfolio will be interviewed about their experience.
Refined guidelines will be made available for those starting their dissertation in September 2011. It is hoped the numbers choosing a dissertation by portfolio will be large enough for a meaningful comparison with L2 students undertaking a traditional dissertation. Also, if there are sufficient numbers opting to develop a portfolio, the student achievement levels will be analysed to compare with the findings of Chang (2008) to see if they can be reproduced for M-level students.

Conclusion
This paper has proposed a structure for an integrated portfolio for use at the dissertation phase of an M-level computing course. The primary purpose of the portfolio is to asses a student's ability to produce a piece of research. It avoids the problem of the student being penalised for their lack of academic English ability while still allowing them to meet the defined learning outcomes.
Initial results suggest dissertation by portfolio is a valid alternative to the traditional written dissertation. Although it is not suggested they be a replacement for traditional dissertations, they should be considered as an alternative, particularly for technically able L2 computing students undertaking more technical projects.