Students´ Perception of Real-Time Quiz Kahoot! As a Review Tool in Higher Education: A Case of Study

majops@unex.es Abstract —The use of gamification has spread in recent years, proving to be an effective tool with which to increase student motivation and create constructivist learning environments. To clarify aspects such as the degree of acceptance and motivation, if the limited time generates stress and when it is more favorable to carry out them, a survey was developed in 48 second-year students of an Agricultural Engineering degree. Additionally, to assess the degree of consolidation of knowledge, 5 questions chosen randomly from 3 previous Kahoot!s relative to three thematic blocks of 2 weeks each were repeated in a review Kahoot! in week 7 (with 5 weeks, 3 weeks and 1 week, respectively, from each corresponding Kahoot!). More than 95% of the surveyed students showed high or very high satisfaction, mainly due to the increase in participation, entertain-ment, usefulness for understanding and reinforcement of knowledge. However, more than 25% of the students felt anxiety, with the limited time to answer constituting a possible cause of this. The effectiveness of this tool for consolidating knowledge was demonstrated, at least until 5 weeks, since the number of questions remembered by the students was greater than 20% in the three questionnaires carried out, independently of the time elapsed between the questionnaire and their review in the final


Introduction
In recent years, in the field of education, student engagement has become recognized as an essential attribute for enhancing motivation and promoting learning. For this purpose, the implementation of gamification, as the use of games in learning, has been generally considered a successful tool [1][2][3][4]. The core idea of gamification lies in the use of the motivational strength of game elements in an educational context [5].
The implementation of gamification has been generally considered as an effective way to create constructivist learning environments [6]. However, there are other aspects that remain less clear. Among them, the evaluation of its impact on learning outcomes at different educational levels, with contradictory results, causing its questioning [5], may be due to the lack of specific assessment tools. Another factor questioned is in relation to when it is more favorable to carry out these games, whether at the beginning of a session or at the end [7], which is also of interest to detect whether the games awarded with points and with limited time generate stress in university students [8].
In this paper, we will attempt to answer these questions. For this purpose, the first part of this paper reviews the existing literature on gamification with Kahoot! at university levels, highlighting both advantages and disadvantages. Afterwards, we describe how the following were determined through a survey: i) the degree of acceptance and motivation that Kahoot! produces in students; ii) if the limited time generate stress; iii) when students prefer to carry out the game. Finally, to establish if Kahoot! helps students to consolidate knowledge, three Kahoot!s were made regarding different thematic blocks with two weeks elapsing between them. From each of them, five questions were selected and repeated in a fourth Kahoot! one week prior to the last. Therefore, the consolidation level was studied after five weeks, three weeks and one week.

2
Review of the Literature

Gamification at university with Kahoot!
Not all educators are creative enough to include gamification in their lessons. For this, there are numerous online tools, such as Kahoot! that offer a variety of excellent options to support active learning [4]. The use of multiple devices, from laptops to mobile phones, which was the resource used in this study, further increases the interest of the students. It was launched by the Norwegian University of Technology and Science in 2013, reaching up to more than 70 million users across the world [9][10].
Kahoot! is a classroom response system (CRS), obtaining instant feedback on every response. The questions that are previously prepared are projected in the classroom, and the students, signed in with a PIN number using a nickname (in this study, it was compulsory to use their real names) with their technological device, answer them in a previously determined time frame. Kahoot! is very versatile, enabling the selection of the type of answer-true-false, simple multiple answer, etc.-to include images and videos, or to configure different variables, such as the available time to answer the questions [10]. After each question, the correct answer is shown, and the teacher can explain each answer, which is where the pedagogical potential lies [11]. Another key point is that, after each question, the game ranks players based on speed and accuracy, and at the end of the quiz, the names of the top five players are displayed on the leaderboard. Results can be downloaded to highlight problematic questions and identify students who may be struggling [10][11]. Additionally, this is precisely, according to [8], where its main handicap lies. The fact that the obtained score is higher if the response is quicker could create anxiety and stress in many students, who, in their desire to obtain better scores, risk too much to the detriment of spending enough time to find the right answer.
In a very recent review on gamification in science education [5], it is noted that, out of 24 studies analyzed, ten were in higher education, using Kahoot! in only three of them. Regarding the positive results, which highlight the increase in motivational outcomes [12][13][14], other studies, such as [15][16][17], have pointed out an improvement in classroom dynamics, an increase in the enjoyability of lectures, the opportunity for guided thinking and review, etc. On the other hand, while the authors of [11][12] did not find significant learning results, and the authors of [13] found that a low score on the scoreboard and insufficient technological skills affect the process negatively, the authors of [4,15] found it to be an effective learning strategy.

2.2
The objectives of the paper The aims of this work were, therefore, threefold: first, to analyze the satisfaction level of second-year Plant Science students undertaking a university degree in Agricultural Engineering, based on their opinions on how the use of Kahoot! has helped them in their learning process; second, to determine which was, in the students' opinions, the best moment at which to carry out the questionnaires, at the beginning or at the end of the class; finally, to quantify whether the use of these questionnaires really helps to consolidate knowledge in the students.

3
Review of the Literature

Gamification at university with Kahoot!
This study was conducted in 2020 between September and November using second-year Agricultural Engineering students at the University of Extremadura, a public university in Spain. The course was Plant Science, and there were 48 participants.

Procedure
During the experimental time, three thematic blocks of two weeks each were taught. At the end of each block, students performed a Kahoot! with 12 multiple choice questions. The two first Kahoot!s were performed at the end of the last session of each block session and the third at the beginning ( Figure 1). In all of them, after answering each question, the wrong options were explained to reinforce the concepts.

Opinion on the use of Kahoot! as a learning tool
To obtain the information regarding students´ satisfaction in the use of this tool in the teaching-learning process, a questionnaire based on the Driscoll Questionnairedesigned in 2012 and used by authors such as [18]-with modifications to focus it on motivation and detecting stress was used. It was chosen in this study because it is based on the selection of the most representative and appropriate items to address the objective on this work. The questionnaire consisted of ten questions graded on the Likert scale, with 5 being totally in agreement/higher score and 1 being totally in disagreement/lower score.
The items of the questionnaire were as follows: 1. I learn the contents in a playful way 2. It encourages my participation 3. Lessons are more entertaining 4. It is interesting as a continuous evaluation system 5. I understand the contents more easily 6. It is useful to explain wrong options after each question 7. I enjoy competing with other players 8. I do not feel anxiety or stress playing Kahoot! 9. The higher score for responding faster does not make me anxious 10. My overall satisfaction with the game.
In addition, students were asked if they preferred to engage in the Kahoot! at the beginning or at the end of a session, to evaluate whether the fatigue of carrying it out afterwards is perceived as a negative factor, or as a positive way to finish a session.

Determination of knowledge consolidation
To evaluate the degree of knowledge consolidation or if significant learning was achieved, students responded to a fourth Kahoot!. This was performed one week after finishing the third thematic block, i.e., with a duration of five, three and one week, respectively. This review Kahoot! comprised five questions randomly chosen from each previous Kahoot! totaling 15 questions (Figure 1).

Data analysis
The analysis of the obtained data after the assessment of the items of the questionnaire was quantitative and based on a descriptive analysis in SPSS 23, using ANOVA and a dependent T-test.

4
Results and Discussion

Analysis of the Kahoot! approach´s acceptability.
The questionnaire can be divided into questions relating to students´ engagement (questions 1 to 7), if Kahoot! generates stress or anxiety (questions 8 and 9) and a final question for overall assessment. Table 1 confirms that the students' engagement is relatively high, with averages for all the items (questions 1 to 7) higher than 4. For the questions related to the development of the lessons and the use of Kahoot! as an evaluation tool, i.e., questions 1 and 4, the results were very satisfactory, with averages of 4.26 and 4.06, standard deviations of 0.72 and 0.76 and a median of 4, respectively. However, these results were slightly lower than those related to the participation, entertainment, usefulness for understanding and reinforcing knowledge and enjoyment. Between these questions, it is worth noting the high rating of the students regarding questions 2 "It encourages my participation", 3 "Lessons are more entertaining" and 6 "It is useful to explain wrong options after each question" with averages of 4.40, 4.71 and 4.60, and a median of 5. However, it can already be seen that a small percentage of students (2.1%) answered questions 5 "I understand the contents more easily" and 7 "I enjoy competing with other players" in a negative way (2/5). In any case, the means are 4.12 and 4.29, with medians of 5 and 4, respectively.
According to [4,[12][13][14][15][16][17], the results clearly confirm the perception that Kahoot! is a useful gamification tool that breaks the monotony and helps students to participate in classes more actively, improves classroom dynamics, makes lectures more fun, allows guided thinking and review, etc. In fact, questions regarding the importance of review, participation and whether the lessons are more enjoyable were answered by more than 84% of the students with answers ≥ 4 ( Table 1).
The results in relation to whether Kahoot! makes them anxious, i.e., questions 8 "I do not feel anxiety or stress playing Kahoot!" and 9 "The higher score for responding faster does not make me anxious", showed a slight anxiety on the part of the students, with responses 1 and 2 (25 and 20.9%, respectively) and a high percentage of neutral responses (45.8 and 29.2%, respectively), giving averages of 2.98 and 3.35 and medians of 3 and 4 ( Table 1). This confirms the prior observation in class that many students, to obtain more points, take risks more than they should. Esteves et al. [8] observed that 15% of students who used Kahoot! felt negatively but did not express why. Despite the large number of studies about Kahoot! almost none of them delve into why a certain percentage of students feel negatively about using it. Therefore, this factor must be studied in the future to attempt to reduce these percentages.
Despite the handicap of time stress, overall, the students rated the use of Kahoot! in university teaching very highly, with 95.9% giving positive or very positive responses, with an average of 4.48 and a median of 5 (Table 1). These results are similar to those obtained by [15], who obtained 87% positivity.

When do you prefer to take the Kahoot!?
Students clearly prefer to carry out the Kahoot! at the end of a session rather than at the beginning of the next, concretely, 92% of students. This can be attributed to the fact that students generally perceive them as a positive way to end the lesson and review the contents of the block with a bit of competition [19] and not as a traditional exam [7]. This aspect has scarcely been addressed in the literature, although the authors of [8] suggested using Kahoot! at the beginning of a session not only to consolidate the contents of the previous class but as a diagnostic test to determine the students' prior knowledge.

Degree of knowledge consolidation
The degree of knowledge consolidation is probably the most controversial aspect of the Kahoot! with positive [4,15], negative [13] or neutral [11][12] results. Most studies that try to determine whether the use of Kahoot! helps to consolidate knowledge by dividing classes into two groups to compare the group that uses Kahoot! with the group that does not. In this case, to clarify the consolidation, this study, using a single group, attempted to measure whether and for how long Kahoot! helps to consolidate concepts. Figure 2 shows that the Kahoot! increased students' effective knowledge learning, as they performed better when the questions were repeated, regardless of whether 5 weeks, 3 weeks or 1 week had elapsed. Concretely, the average score obtained in the 5 questions that were after repeated in the review Kahoot! increased by 27% in thematic block 1 in comparison to Kahoot! 1, although 5 weeks had elapsed between the first and the second, by 20.1% in Kahoot! 2 and by 25.6% in Kahoot! 3. We understand that this way of assessing consolidation is different from that in other studies, but the results can be considered as very satisfactory, as the increase in score was higher than 20% in all the tests.
As [5] found in a very recent review, all the studies that displayed an increase in motivation also reported increased motivation, engagement and learning outcomes. Moreover, the different durations elapsed between questionnaires allow us to answer one of the main drawbacks pointed out by [20] regarding concepts embedded in gamified environments that are often.

Conclusions and Future Work
Though this study has some limitations (a small sample, short duration, etc.), it provides results implying that gamification can ensure the engagement of students, promoting students´ active participation and eliciting effective learning in the medium term. Additionally, it is important to clarify that part of this work was based on stu-dents´ opinions about how the use of Kahoot! has helped them during their learning process. The results show that the overall satisfaction of more than 95% of the students surveyed is positive or very positive, mainly regarding the positive perception regarding the participation, entertainment, usefulness for understanding and reinforcing knowledge. However, despite this positive result, in more than 20% of the students, it caused anxiety, with the limited time to answer constituting a possible cause of this. On the other hand, more than 90% prefer to carry out the Kahoot! at the end of the session instead of at the beginning of the following session as a fun way to end a lesson. Finally, its effectiveness for consolidating knowledge was shown, at least over until 5 weeks. The number of answers that the students remembered was greater than 20% in the three questionnaires carried out, independently of the time elapsed between the questionnaire and their review in the final Kahoot! Although the findings from this study are consistent with previous research, further research into this area is required to corroborate the effectiveness of gamification as a strategy for increasing student engagement and effective long-term learning. For this purpose, new experiments should be carried out with more students and a longer duration and more attention given to the anxiety caused by the limited time for answering.