Elsevier

Computers & Education

Volume 87, September 2015, Pages 98-111
Computers & Education

The benefits and challenges of large-scale deployment of electronic voting systems: University student views from across different subject groups

https://doi.org/10.1016/j.compedu.2015.04.004Get rights and content

Highlights

  • A mixed-method approach based on non-linear principal component analysis and content analysis of survey data was used.

  • Students from Health and Human Sciences experienced more learning benefits and fewer challenges compared to other groups.

  • The subject differences were attributed to varying levels of summative assessment and frequency of use.

  • The students' views on usability of EVS were divided, despite EVS scoring high on the Simple Usability Scale.

  • The less positive views on usability emerged when EVS was used for summative assessment.

Abstract

Electronic Voting System (EVS) is a classroom technology that provides a means to increase students' engagement, attention and attendance. The purpose of this paper is to provide a deeper insight into students' views on the benefits and challenges of EVS in the context of a large-scale institutional deployment and across different subject areas in higher education.

The data were collected from an online survey of 590 students across eleven academic schools at a UK university.

The non-linear principal component analysis of 32 question items from the survey showed that learning benefits, classroom-related benefits, usability and student-centered challenges are four distinctive dimensions in student's perceptions of the use of EVS. The non-parametric group comparison tests suggested that there are significant differences in learning benefits and challenges across different subject groups. However, the disparity appears to be related more to the way the EVS was used and the experience of students with it, rather than resulting from disciplinary differences. Content analysis of open questions revealed that summative use and staff competencies are the main issues related to EVS use by students. Finally, despite the overwhelming perception of the ease of use, it was found that usability could be an issue for students when EVS was used for summative assessment.

The implications of the study are: for practitioners, it underlines the importance of the focus on formative benefits of EVS as only then and regardless of disciplinary differences, can the promised rewards of the technology be gained; for institutions, it outlines some of the new challenges specific to the large-scale institutional implementation, judged through the lens of students' experience; for researchers, it provides an overview of the literature on large-scale deployment of EVS and it suggests some new areas for research on the use of EVS in higher education.

Introduction

Since their introduction in the late sixties (Kay & LeSage, 2009:819) Electronic Voting Systems, also known inter alia as “clickers”, Audio Response Systems, or Personal Response Systems are being increasingly used in university education as a means to increase students' engagement, particularly in large classes. This is commonly achieved via in-class quizzes based on multiple-choice questions (MCQ), which can offer formative or summative outcomes. The technology behind EVS is very simple, consisting of a receiver, used for collecting the question responses and a number of handsets used for sending the responses by individual users. The examples of use cited in the literature are largely focused on small-scale implementation, within a single module and typically within subjects such as science, technology, engineering and mathematics (STEM). Using a mixed-methods approach, this paper seeks to address the gap in the literature and to explore the large-scale institutional implementation of EVS and differences in use and outcomes across academic disciplines, including those outside the STEM subjects.

The research presented in this paper is part of a wider evaluation project initiated in the 2011/12 academic year that examined benefits and issues of large-scale institutional deployment of EVS technology for assessment and feedback at a medium-size post-92 UK university (UH) with in excess of 27,000 students. The aim of the project was to extend the benefits of EVS experienced by the early adopters from the area of Health and Human Sciences, across the whole university, while increasing the efficiencies related to the management of the handsets and the staff workload. The latter was achieved through an increase in the use of EVS for summative assessment. The central implementation campaign included regular staff training, installation of receivers in all teaching rooms, purchase of additional handsets, and the standardisation of operational procedures including distribution, ownership, and cost of handsets. By September 2012, 7265 EVS handsets had been purchased, 35% of the classrooms had been equipped with the EVS technology and 114 additional staff members i.e. 9.6% of the total staff population had been trained on technical and pedagogical use of the EVS.

The aim of the evaluation project, which worked alongside the institutional deployment project, was to support a scholarly approach to the changes in practice, while the purpose of this paper is to unravel some of the tacit and less explicit findings supported by the further analysis of the existing data. It is important to emphasise that although the data used for this paper are a subset of data collected during the evaluation project, they were collected purposefully in order to answer specific research questions, in an attempt to address some of the gaps identified in the current literature.

Section snippets

Literature review

The literature on EVS in university education dates from the 1990s (Dufresne et al., 1996, Mazur and Hilborn, 1997) to the present day and includes seven literature reviews conducted between 2002 and 2014. Amongst these, the review paper by Kay and LeSage's (2009) remains the most systematic and comprehensive appraisal of the literature, which has provided a framework for research in this area.

Research questions

This research accordingly addresses some of the issues and gaps reported in the literature review above and considers the following research questions:

  • 1.

    What are the main components contributing to the students' views on EVS in a context of a large-scale institutional deployment?

  • 2.

    What are the new student-centered challenges that arise in a large-scale institutional deployment of EVS?

  • 3.

    What is the impact of increased summative use of EVS on students' views?

  • 4.

    What are the differences in usage and

Methodology

This research is based primarily on the gathering of data through a purpose-designed student survey (N = 590) with some additional comments provided by students studying at UH in 2011/12 via their online reflections. In the latter case, the students (N = 26) captured their experiences of using EVS in class by either a daily written blog or by regular audio or webcams recordings. For the purpose of this paper these comments are used to illustrate the findings from the survey in the related

Data analysis and results

The first part of this section includes the rationale for using the specific data analysis techniques in relation to the research questions.

To answer the first research question (“What are the main components contributing to the students' views on EVS in a context of a large-scale institutional deployment?”) the Principal Component Analysis (PCA) technique was used (Section 5.1).

PCA is a technique used for reducing a number of variables into a smaller set of “principal components” representing

Discussion

The goal of the paper is to offer a deeper insight into students' views on the benefits and challenges of EVS in the context of a large-scale institutional deployment and across different subject areas and different levels of summative and formative use. This section includes the discussion of the results and limitations of the study and it is structured according to the initial research questions.

Conclusions

The benefits of EVS for students' learning, engagement and motivation are well-known from the numerous studies on small-scale implementation. The aim of this research was to obtain a deeper insight into students' views on the benefits and challenges of EVS in the context of a large-scale institutional deployment and across different subject areas, the topic that is currently under-represented in the EVS literature.

The results of this research suggest that the scale of EVS deployment can create

Acknowledgements

The research was partially supported by the JISC Assessment and Feedback Programme Grant Funding 5/11. We thank Neil Spencer and the anonymous reviewers for their useful comments on the draft of this paper.

References (34)

  • J. Brooke

    System Usability Scale (SUS): A quick-and-dirty method of system evaluation user information

    (1986)
  • D. Bruff

    Classroom response system (‘clickers’) bibliography

    (2014)
  • J.E. Caldwell

    Clickers in the large classroom: current research and best-practice tips

    Life Science Education

    (2007)
  • D. Cardillo

    The use of clickers in the law school classroom

    The Law Teacher

    (2008)
  • R.M. Carini et al.

    Student engagement and student learning: testing the linkages

    Research in Higher Education

    (2006)
  • L. Cohen et al.

    Research methods in education

    (2000)
  • R.F. DeVellis

    Scale development: Theory and applications

    (2012)
  • View full text