Identifying opportunities and gaps in current evaluation frameworks – the knowns and unknowns in determining effective student engagement activity

Abstract The higher education sector is under increasing pressure to deliver more and to evidence the fruits of its contribution to society. There is a particular focus on the enhanced engagement of students in order to maximise success, and for this engagement to be achieved equitably across the student body, regardless of demographic backgrounds. This calls for the sector to increase its capability to enhance student engagement in a targeted manner, and to be able to evidence success. The purpose of this article is to critically review, as a result of trialling, frameworks which offer guidance on the structuring of student engagement activity and/or encouraging behaviours associated with student engagement whether at student, teacher or institutional level. We assess the value of these frameworks in practice by identifying opportunities provided by them, reveal gaps in how they support effective student engagement activity, and identify further work for the sector in plugging the guidance gaps that lead to sub-optimal evaluation.


Introduction
The global growth of higher education has led to increased overall cost and a desire for universities to deliver 'more, more explicitly, and to deliver demonstrable quality' (Stensaker and Sweetman 2014, 251).Enhanced student engagement, defined as including time on task, social and academic integration, and teaching practices (Kuh 2009), is one way to deliver more.Student engagement is associated with increased student activity (Kahu 2013), better retention and completion rates (Thomas 2012), and greater work readiness (Department for Business Innovation and Skills 2015).Put simply, higher student engagement leads to better retention and better grades (Kahu 2013), and it is increasingly seen as pivotal as a means to deliver social mobility by creating the conditions for positive engagement by all students including from disadvantaged backgrounds (Universities UK and National Union of Students 2019).This positioning of student engagement suggests it is not just one of many useful factors to engender, but rather that it is foundational to student success (Wilson 2018).
As a result, learning institutions need to know much more about how and why interventions work (Brooman and Darwent 2014).The question of what evaluation is appropriate for measuring Student engagement; evaluation frameworks; equity; attainment; student outcomes student engagement is especially challenging as the concept is poorly specified (Kahu (2013), wide ranging in size and extent (Kuh 2009) and is 'fuzzy' (Vuori 2014, 517) because it involves complex interplay between student factors, such as educational background, and institutional and structural factors (Bryson (2015).
The purpose of this article is to contribute to the body of knowledge about evaluation in higher education by critically reviewing candidate frameworks which offer guidance on structuring student engagement activity and/or encouraging the behaviours associated with engagement whether at student, teacher or institutional level.Our evaluation of the frameworks includes a specific judgement on their suitability for equity and attainment work.

Student engagement as a desirable construct
Student engagement results in increased content knowledge, skills, competencies and personal development (Brooman and Darwent 2014;McGrath et al. 2015).There is increasing interest in evaluating and improving the amount, type and quality of effort that students invest in their studies (Kandiko Howson and Buckley 2017).Higher education providers have a role in developing student engagement by designing curricular, co-curricular and extra-curricular opportunities to build students' capacities to engage in learning and to develop a sense of belonging between students and teachers (Thomas 2012).
The efforts of higher education providers to create engaging learning are not experienced equitably.Students from more privileged backgrounds enrol in more selective universities and get better results, even when prior academic attainment is factored in (Crawford 2014;Crenna-Jennings 2018).White students achieve better degree outcomes than their black counterparts (Thiele et al. 2016).The 'degree dividend' is smaller for ethnically diverse students, reducing their life opportunities and long-term careers (Office for Students 2019a).While student characteristics, such as cultural capital, may account for some of these differences, it is exacerbated by structural factors (Bryson 2015), such as failing to prepare lecturing staff to teach diverse students (Naylor and Mifsud 2020).Despite the focus on these differences in degree awards, progress has been modest and evidence to back up the effectiveness of interventions to enhance student engagement equitably appears limited (Robinson and Selvestini 2020).

Why evaluate?
Evaluation practice serves a number of purposes.It can be undertaken for accountability or development (Newburn 2001); it can be mandated by funders (Mayhew 2011) and can use a wide range of approaches and activities, making an agreed definition difficult (Stufflebeam and Shinkfield 2007).In higher education, evaluation in the form of monitoring programmes and student behaviour, with explicit indicators and measures of success, is an essential responsibility of university leadership (Thomas 2012).The Office for Students (OfS), the regulatory body for England, has recently required providers to "prioritise generating, sharing and learning from evidence about what is working to make higher education more equitable" (2019b), and has set up a centre for Transforming Access and Student Outcomes (TASO) to identify best practice on evidence-based evaluation (Office for Students 2021).A specific challenge in evaluating student engagement is that engagement practices occur in myriad 'messy' settings (Wilson and Dauncey 2020, 19), compromising some of the strict methodological ambitions of evaluation (Maruyama 2004) The methods deployed in evaluation reflect underlying stances about the nature of knowledge (Broughan and Prinsloo 2020).Current ideas about knowledge focus on evidence-informed processes, quantitative data and clear causal relations between activity and outcome, and lack the voices of some key stakeholders, such as the students themselves (Foster and Francis 2020;Wilson and Dauncey 2020).Involving students in understanding their engagement has the potential to enhance agency (Francis et al. 2020), empower students (Broughan and Prinsloo 2020) and thus be an enhancement instrument in its own right.
Finally, there are questions regarding what the various indicators and data available to higher education represent.Causality between indicator and outcome is often simply assumed (Foster and Francis 2020), and intermediate and proxy indicators of student success or satisfaction often struggle to be seen to directly impact on the longer-term outcome measures sought by stakeholders (Wilson 2018).

Materials
We invited practitioners using evaluative frameworks in the RAISE network to get in touch via an open call (RAISE -Researching, Advancing and Inspiring Student Engagement -is a UK based but worldwide network of staff and students in higher education).The intent was to identify frameworks or guidelines that are effective across institutions while maintaining a contextual understanding of what makes engagement special and effective.We drew up criteria to assess responses to this call, i.e. that candidate guidance must have a prima facie capability to evaluate a range of student engagement activities, have been used in practice, and its users willing to participate in a collaborative critical review.Potential uses of frameworks included any areas where students were 'being engaged' .This focus disqualified those which are more suited at shaping the understanding of the context and nature of student engagement and partnership working (e.g.Healey, Flint, and Harrington 2014).Each of the four selected frameworks is described in use in a university setting and focussed on equity work, in that it is open to all students but has the potential for narrowing the gap in the experience of disadvantaged students.Our individual experiences of and reflections on the application of these theories act as the materials for our analysis.
The Student Engagement Evaluation Framework (SEEF), was developed to champion and develop student engagement and support practitioners in evaluation (Thomas 2017).It focusses on structuring the steps of evaluation and prompting reflection on relevant factors that may be overlooked (e.g. a diverse student population) through the use of prompt questions and a structure which centralises consideration of multiple groups.It also prompts evaluators to think about data and other evidence (e.g.pre-existing outputs) and tools that could be used to inform an evaluation.SEEF encourages evaluators to account for a multi-faceted concept of effectiveness in student engagement relating to participation, experience and longer term impacts (see Table 1).SEEF was used at the University of Portsmouth to evaluate the process of developing a new staff-student partnership initiative.
Portsmouth's 'Digital Education Ambassadors' were recruited to undertake projects to advance, promote or evaluate Technology Enhanced Learning across the institution.Example projects include the partnership development of podcasts about undergraduate research projects, and video guides for using technology.Student involvement ranged from recruitment and training of student participants to the name of the scheme itself.This scheme was cut short due to Covid-19, leaving few opportunities for the planned data gathering for evaluation.Instead, SEEF was used by the project lead to explore options for an improved roll out in future iterations.
Kahu's Conceptual framework of Student Engagement (2013 version) was used at the University of Leeds to evaluate engagement in the School of Earth and Environment (this framework was later updated to incorporate the 'educational interface' by Kahu and Nelson (2018)).The framework (Kahu 2013) was designed to capture the key factors likely to influence engagement bringing together four distinct but overlapping perspectives of student engagement that were observed in the literature: behavioural, psychological, socio-cultural and holistic.The resulting framework places the student at the centre, acknowledging the role of structural and psychosocial influences.Student background is a fundamental consideration and highlights the 'unique nature' of each student's experience (see Figure 1).
The framework was used in the School of Earth and Environment (Dyer, Jackson, and Livesey 2018) to structure the data collection from focus groups carried out with students across six programmes and four years of study.The students were asked to note keywords relating to 'student engagement' or what it looks like to be 'engaged' on sticky notes, and these were then clustered and explored through group discussion.This process was repeated with 'enablers of engagement' and 'consequences of engagement' .This was a valuable process for showing what was working well in the School and what was not.For example, a weekly bulletin highlighting student-relevant events was widely praised and an induction week field trip was cited as valuable for building friendships.However, the lack of a sustainability-focused student society was seen as a barrier to engagement.
The Student as Partners (SaP) Framework (Bovill 2017) was used as a planning tool with academic staff and student curriculum advisers (students employed to develop staff-student partnerships) at the University of Brighton.The SaP framework is based upon a 'ladder' of participation (Bovill and Bulley 2011), which sees staff-student partnerships progressing from students being informed, consulted, participating, through to working in partnership and being in control.The Brighton version of the framework was developed as part of an initiative which encouraged staff and students to develop small-scale projects that would impact on the curriculum.The Brighton framework was modified from Bovill's original, with 'action research stages' replaced by 'project phases' (see Table 2) to align it more closely with the project plan form.Students and staff populated the framework as a learning activity during partnership training sessions to set out the type of partnership at each stage of the project.This contextualised a range of  activities, for example discussions on active listening and the joint completion of project plan form.
Completing the framework allowed what could appear to be very small projects with minimal impact on student learning to be linked into a series of activities (see Table 2), starting for instance with consultation with all students in a cohort, through a project involving a sub-group of these students, rounding off with dissemination back to students.In terms of capturing student engagement it shows a quantifiable number of students involved in initiatives.The action of completing a framework can form an important bridge between student consultation, often cited as weak or tick box exercises (Webb, Jarnecki, and Russell 2014), and the more pedagogically powerful engagement of students as partners.
The Behaviour Change Wheel (BCW), a framework for designing, developing and evaluating behaviour change interventions generally (Michie, van Stralen, and West 2011), was used by a Midlands consortium of universities and colleges running and evaluating a range of interventions to address barriers to student engagement for disadvantaged students.The BCW was developed to cover the range of intervention options available across a range of behaviours in different sectors.
The BCW has three layers (see Figure 2).The inner core features three determinants of behaviour that help intervention designers understand 'what needs to change' (Michie, Atkins, and West 2014, 57).Surrounding these is a middle layer of intervention functions or types, and seven policy categories make up the outer layer.A key feature is that the relationships across the BCW recognise that specific behaviour might have multiple determinants, addressed via multiple interventions and policies.The BCW was used as to conceptualise and evaluate the OfS-funded DRIVER project, which focussed on developing a bank of successful activities to enhance the experience of students entering higher education from colleges, where students from disadvantaged backgrounds are highly represented.The eight partners used the BCW to re-think student engagement activity as formed of a series of behaviours, whether at the institutional, teaching or student level, which support the student to respond by enhanced engagement (the collaborating partners were Coventry

Methods
The frameworks reviewed in this paper are all theoretically informed and are examined in an applied setting.As such this is categorised as translational research, as it tests the usefulness of such frameworks in addressing real world problems (Salovey and Steward 2004).We approached the problem as action research, positioning ourselves as practitioner researchers, with our inquiry focussed on our professional practice and experiences, with a goal to provide increased utility and effectiveness for fellow practitioners (Parsons and Brown 2002).The project was approved by University of Portsmouth's ethics procedure.A two-step method was used to assess the frameworks.The first step was to review each framework for its performance through the linear stages of an evaluation.The second step was to agree on a series of questions we would respond to, in order to assess the merits of each framework and compare strengths and weaknesses in the context of student engagement in higher education.

Data analysis and interpretation
This section reflects our iterative analysis of the 'results' of using the frameworks and our discussion on the meaning of these experiences against expectations or alternatives.
The first step in structuring our review was to follow the linear steps of an evaluation: preliminary thinking, evaluation design, implementation and analysis, and reporting and using evidence (Thomas 2017).The next step was to open up our analysis and discussion by reflecting on what questions practitioners might ask when choosing a framework for an evaluation.We generated several questions and then mapped them against each other to distil them into five: • How useful is each framework for aiding understanding of the process of conducting an evaluation?• How ready, straightforward and complete is each framework for implementation by users of varying experience levels?• Is the guidance relatable to the higher education context?• How well do the frameworks respond to evaluating student engagement activity focussed on making higher education more equitable?• What are the methodological underpinnings?

Utility of each framework in aiding understanding the process of conducting an evaluation
The key benefit of using the four-stage review is that enabled us to identify the relative merits of each framework (see Table 3).
The Kahu and BCW Frameworks are strong at the conceptual phase ('preliminary thinking' and 'evaluation design') and set evaluations on track by focussing on barriers and the identification of optimum ways to overcome these.The BCW was designed to address the common practice of designing interventions based on 'personal experience, a favoured theory or cursory analysis' (Michie, Atkins, and West 2014, 14), and instead focus on thorough assessment of the behaviour sought and its antecedents.The BCW and Kahu frameworks become less prescriptive and offer less advice as the stages of the evaluation progress.However, if the initial steps are observed rigorously, questions of what data to analyse, and the generation of appropriate reports should follow naturally.In contrast the SEEF focuses less on intervention design and starts instead with a requirement to specify exactly what is to be achieved and indicators of success, ensuring alignment between the purpose of an activity and its evaluation (see Table 3).It also encourages specification of the data, evidence and tools to be used, and while this exactness aids communication with audiences, it runs counter to the 'messiness' of student engagement practice.
SaP is rather a different type of framework in that while it has a linear structure focussed on the steps of planning, implementation, observation and reflection, its focus is equally on who is involved and in what capacity.This is potentially an advantage in ensuring the barriers to change are identified by deep involvement of all those involved, but this is at the expense, as can be seen in Table 3, of detail on how and what is required at these different stages.

Readiness for/ease of use by a wide range of participants
The BCW is not designed to be a complete guide to the evaluator as the step-by-step guidance stops at the intervention implementation (Michie, Atkins, and West 2014).In the DRIVER programme, advice for the phases beyond this was drawn from use of a logic model, which links the planned action to its expected results (Frechtling 2007).The target audience for the BCW is 'intervention designers' but also includes those interested in 'systematically applying theory and evidence to designing and evaluating behaviour change interventions' (Michie, Atkins, and West 2014, 13).In practice, behavioural science expertise was needed to provide specialist support in the codification of interventions and identification of the determinants to be changed (Wilson, Broughan, and Gakhal 2018).The Kahu framework is a candidate for a comprehensive evaluation as long as there is scope for qualitative data collection.It can be put to use for evaluating existing initiatives, for predicting the outcomes of planned initiatives and for developing initiatives based on desired outcomes or consequences of engagement.In contrast SaP -with its focus on partnership -was found to be an incomplete evaluation tool through giving no guidance on the evaluation stages of preliminary thinking or reporting and using evidence (Table 3).Its simplicity means that it can be picked up and used readily, but this same factor also limits the extent of its usefulness.Simplicity is a positive and negative also for the SEEF.Its simplicity, prompts and focus on process makes it a good resource for a novice evaluator, though it lacks methodological guidance.More experienced evaluation teams might want to consider the SEEF for the practicality of framing an evaluation (e.g.what data sources to choose) in combination with the ideas (but less direction) that underpin the other three frameworks.

Is the guidance sufficiently relevant to the higher education context?
The SEEF is designed to be used in higher education and is thus aligned to familiar language and impact indicators such as 'improvements to continuation, completion, attainment, satisfaction, employment' (Thomas 2017, 5).Similarly, Kahu's framework is designed for use in higher education, but to achieve its full potential it requires the generation of rich, qualitative data to really be able to explore the antecedents of change.It requires in-depth analysis of different points of view, of areas which can be generalised and those which are nuanced according to the student.
SaP has its antecedents in community partnership and was used in a local-level setting where it was found to work well on planning proximal changes.It is less likely to be a choice when working at tackling institutional-level and distal targets.In contrast, the BCW is broadly designed, although it has been identified as potentially useful in higher education (Wilson, Broughan, and Marselle 2019).It has utility for larger institutional level evaluations looking to reveal the effective components of interventions.Use alongside the SEEF, which is more prompt-oriented, would further help to align to a higher education setting.

How well do the frameworks respond to student engagement activity focussed on making higher education more equitable?
The SEEF prompts the evaluator to consider various elements which could be related to equity and attainment via suggestions of potential metrics associated with student engagement, including factors they may otherwise overlook, such as how and why indicators of engagement may be different for some groups of students.Considerations of equity are therefore implicit in the framework, but this does not extend to the level of suggesting the inclusion of specific groups or contexts.
By keeping the individual and unique experiences of the student at the heart of the Kahu framework, it is possible to identify a range of ways to overcome barriers, explore the myriad forms engagement takes, capture ways students define engagement and determine what consequences will affect different groups.In focussing specifically on 'lifeload' as part of the antecedents, this framework is a good choice for exploring equity.All of this information taken together is invaluable for informing specific student engagements as well as developing more generalisable guidelines.
Similarly, the BCW's focus on barriers as being a function of student opportunity, as well as motivation and capability, helps focus on the weaknesses of institutional structures, making it highly suitable to identify the reasons for differential engagement.It offers guidance on the relationships between barriers and mechanisms to address these, helping to direct choice of the intervention most likely to be successful.
The SaP framework has a high equity potential as it is about empowering voices and sharing of decision-making.While the project at Brighton attracted a diverse range of students, this is not often the case with extra-curricular activity (Lowe and Dunne 2017), and addressing this does not feature explicitly in the framework.

What are the methodological underpinnings?
The cultural preconceptions and values that underpin the different frameworks should also be considered in choosing an evaluation framework.Evaluation is not value neutral and is done to achieve a particular end.BCW was well suited to evaluating an outcomes-focussed and externally funded project.In contrast the SaP framework is focussed on encouraging true participation and creating a situation where everyone who is affected by the issue is equally involved in finding solutions, addressing the concern that those who are the subject of change are frequently not included as research participants (e.g.Jackson et al. 2014;Gainforth et al. 2016).Values about visibility and inclusion are prioritised over collecting more in-depth data and ensuring consistency between evaluation points.
One value embedded in the SEEF Framework is giving people not traditionally involved in evaluation the tools to participate.The aim is to 'support student engagement practitioners in institutions and students' unions to evaluate the impact of their work' (Thomas, 2017,3).Interestingly, including students is not a feature of the framework, as the outcomes-driven approach does not focus on process.The Kahu framework is also open to a wide range of users, and the bottom-up nature of the way the framework is used focusses on inclusivity and participant voice as helping shape the qualities of the evaluation.This openness lends itself to being used by diverse groups of staff and students.

Conclusion and future research
This article responds to the increased focus in higher education on enhancing student engagement as a means to make university teaching and learning maximally effective for the student, the university and society.We aim to contribute to the understanding of appropriate evaluation by reviewing candidate frameworks to identify the opportunities offered by them in practice, to highlights gaps and propose improvements.
From a process point of view, the BCW and Kahu frameworks are focussed on the preliminary stages -Kahu's framework prompts a thorough assessment of antecedents, especially for a higher education context, and BCW assists understanding of the relationship between antecedents and suitable actions.SaP is strongest at identifying stakeholders and their roles, and is soundest for the middle stage of evaluation planning.The SEEF is useful for a non-specialist as a result of its comprehensive series of sector-specific prompts, and is the only framework which offers guidance on analysis and reporting phases.
From a methodological point of view, top-down funder-led outcome evaluations will be well served by the SEEF, the BCW and to a lesser degree, Kahu.Smaller scale interventions, where stakeholder involvement is itself an outcome, might be well served by using the SaP framework.For instance, enhancing a student's sense of community via a partnership approach, used as a means to address satisfaction levels, is more likely to make students feel involved, which is an outcome in itself and likely to lead to increased satisfaction.However, feeling involved is only a proxy indicator of the type of impact expected to be demonstrated by student engagement activity.
We found that the frameworks varied as to their ease of use.The SEEF contained straightforward guidance useful to non-specialists, whereas the BCW was complex for the non-specialist.Which framework to choose therefore depends upon availability of confident users, the purpose or hierarchy of activity, the audience for the results, and the extent to which there is interest in the process of change as much as in longer term outcomes of change.
Future frameworks or guidance should consider explicitly the underpinning views and values behind methodological choices -such as the importance of comparability between evaluations, whose voice counts, and the extent to which engagement in decision-making is an outcome in itself.Consideration of these factors will guide framework choice but are missing from the frameworks themselves.
This work has revealed the need for further work in both guidance and practice.Our literature review noted the sector move towards evidence-based top-down approaches which tend to exclude the research subjects, a troubling turn for research which attempts to measure engagement.While use of the SaP and to a lesser extent Kahu frameworks are ways to include students, a current omission is how top-down evaluation that focuses on cross-institutional comparability aligns to local 'bottom-up' stakeholder approaches to evaluation.
The challenge of isolating the effects of discrete interventions in a messy real-world environment of multiple interventions and changing student cohorts is another area for further work.None of the frameworks address this, yet as it was identified as an issue in all four case studies.We acknowledge that a limitation of our work is our focus on a single case study of each framework.More case study analyses may have unearthed richer observations.
Finally, there are gaps in linking the intermediate indicators sought by some projects, e.g. the 'feeling' of being engaged, or positive student satisfaction, to the longer term achievement outcomes sought from the sector.This also reflects the problem that intermediate and proxy indicators of student success or satisfaction often struggle to be seen to directly impact on longer term outcome-type measures.More evidence of the relationship between intermediate and long-term indicators would give confidence in the causal chain from enhanced experience, through to engagement and success.This would also confirm the continued value in evaluating engagement as an indicator of longer-term outcomes valuable to students and society.
Fiona Handley is Principal Lecturer in Learning and Teaching, in the Learning and Teaching Hub at the University of Brighton.She works in digital learning, and is particularly interested in partnership approaches to enhancing digital capabilities.

Figure 2 .
Figure 2. the Behaviour change Wheel by michie, van stralen, and West (2011) is licensed under the creative commons Attribution license 2.0.

Table 2 .
Bovill and Bulley (2011)on version of the Bovill Framework, inspired byBovill (2017)andBovill and Bulley (2011), with example content from a project in the school of media.

Table 3 .
summary of how the frameworks relate to evaluation stages advocated by thomas (2017).