Making linear multimedia interactive: questions, solutions and types of reflection

Typically, the format of online physics educational videos is closer in style to direct instruction rather than being interactive. One simple way of making videos more interactive is asking questions in one video, providing solutions in the second and seeking student reflection. This study utilises two physics education videos by Derek Muller (Veritasium) which were deployed with 508 first year physics students at The University of Sydney. The students watched the first video, which asked four physics questions, and were prompted to write down their answers. After this, the students were shown the solutions video and asked whether they changed, or did not change their answer, and their reasons for doing so. Data were analysed with a mixed-methods approach. While students whose answers did not match the solutions mostly modified their answers, a nontrivial percentage did not. An interesting finding is that the reasons provided for modifying or not modifying their answers are similar, or ‘mirrors’ of each other. We comment on the nature of the questions and the opportunities for learning by incorporating reflective thinking, particularly through videos that use a question, solution & reflection style.


Introduction
As flipped, blended and online learning increases in popularity, so does the ubiquity of multimedia as a learning tool, especially linear videos, (Davis 2013, Hill et al 2015, Yeung et al 2016. Asking students questions is also pervasive, occurring face-to-face, in exams, and online (Tune et al 2013, Brualdi Timmins 1998, Hargie 1978. Combining the question, solution & reflection framework with online educational video takes a typically linear, non-interactive medium and makes it interactive. Students could be asked to provide their answers as the video pauses, scaffolding student learning, or to provide their answers at the end and compare with the solutions provided (Mayo et al 2009). While comparing, students reflect on their answers (Dewey 1933, Rodgers 2002. Do they notice that their answers are the same as the solution, or not? How do they reconcile their thinking, deciding whether to modify their answers or not? This process, reflective thinking comes in different types and is regarded as an effective mechanism to aid learning (Mezirow 1998, Hubbs and Brand 2005, Lebedev and Sharma 2019. However, the process of reflective thinking is under-researched in the context of online physics education videos.
This study attempts to probe reflective thinking with first year university physics students using question, solution & reflection style online educational videos. The first research question is quantitative, seeking to obtain a sense of the decisions students made. RQ1: How many students modified, or did not modify their answers with respect to whether their answers to questions shown in a video matched/did not match the solution provided in another video?
The second research question is qualitative, probing how students reconcile their thinking justifying their decisions.
RQ2: What reasons do students give for modifying their answers or not modifying their answers?

Multimedia
Multimedia as a learning tool can be incredibly effective, but, like any tool, it must be appropriately designed and deployed (Stelzer et al 2009, Mayer 2005, Mayer and Moreno 1998, Muller and Sharma 2005. There are many flavours of educational multimedia, ranging from video and interactive demonstrations to augmented and virtual reality (Perkins et al 2006, Wieman et al 2010, Steiff and Wilensky 2003, Marçal et al 2018, Psotka 1995, Bower et al 2014, Lee and Sharma 2008. Several well-known pedagogical ideas can help create effective multimedia, such as Socratic dialogue, interactivity, addressing misconceptions, and asking questions (Rose 1999, Perkins et al 2006, Muller et al 2008, Bartels and Kulgemeyer 2018. Asking questions, in particular, is one of the hallmarks of reflective thinking (Dewey 1933).

Reflection
Dewey outlines reflective thinking as a meaning-making process that moves a learner from one experience into the next with deeper understandings of relationships; providing a thread that makes continuity of learning possible. It is a disciplined way of thinking, requiring attitudes that value the personal and intellectual growth of oneself and of others. It is interesting to note that the notion of 'thinking' is retained, making the process intentional, one that can be learnt and consciously practised. Since reflection can be learnt, it can be 'taught' in the sense that learning contexts can be created for reflection to be learnt (see for example Hubbs and Brand 2005). Such learning contexts would, according to Rodgers (2002) interpretation of Dewey, have different phases of reflective thinking: • An experience, and the spontaneous interpretation of that experience.
• The articulation of the problem or the question that arises out of the experience.
• The generation of possible explanations for the problem or question.
• These explanations need to be examined and tested.
Most researchers concede that reflection is a complex, self-referential and organic process (Rogers 2001). To make the study of reflection possible, we use Rodgers' interpretation of Dewey's phases of reflective thinking (Rodgers 2002, Dewey 1933. The complexity of reflective thinking emerges in Mezirow (1998) research which shows that adult learners, such as university students, exhibit different types of reflection. Pertinent to this study are Mezirow's overarching types of reflection, outward focusing and inward focusing. In outward focusing, learners are intent on reconciling their answers with discipline content and tools, while in inward focusing, they are intent on reconciling their answers with their own learning and actions (Lebedev and Sharma 2019). The process allows learners to recalibrate their learning (Sharma and Bewes 2011). This study probes the types of reflective thinking amongst first year university physics students as they interact with question, solution & reflection style online educational videos.

Context
The study was conducted in the first semester of a first-year physics course covering mechanics, thermodynamics, and waves at The University of Sydney. This course consists of lectures and tutorials, and laboratory classes with an online 'pre-lab' activity completed outside of class time. The 'pre-lab', administered through the learning management site Canvas, is accessible for a week. Our study ran as a 'pre-lab' activity aligning with the lectures on mechanics. The activity was not graded but participation contributed 0.125 percent towards students' final mark.

Procedure and intervention
To complete the activity, students logged into Canvas where a link took them to a secure university platform (RedCAP) in accordance with university human ethics research protocols. Figure 1 shows the four components to this activity: (a) video one containing four questions, (b) part one of the survey, followed by a page break-students could not go back, (c) video two containing solutions, and (d) part two of the survey. While the activity was open for a week and students could take a long time to complete the activity, they were allowed only one attempt.

Page one-questions video
The questions video is from the YouTube channel 'Veritasium' by Dr Derek Muller. It is an embedded YouTube link which can be paused and replayed. The video presents a sequence of four different situations, primarily on mechanics, positing a question for each, see table 1. While the situations and the questions vary in style and content, they are perplexing; inviting explanations in line with Dewey (1933) phases of reflection. Figure 2 illustrates how different parts of the video follow the first three of Dewey's phases of reflection in question one; an experience is shown, there is a pause for the viewers to process the experience, the question is posed, and viewers are requested to generate possible explanations. The structure of the intervention: page one contains the questions video and the first part of the survey, followed by a page break; page two contains the solutions video and the second part of the survey. The two pages are separated by a page break that prevents students from going back.

Question number and its focus
What was asked?
Question one A cylinder is shown to roll down an inclined plane. Focus on concepts such as: It rolls, then stops, then rolls again. rotation,inclined plane, What is inside the mystery cylinder? centre of gravity, buoyancy Question two A bicycle has a string attached to the back wheel. Focus on concepts such as: The bike is gently supported so it does not tip over. rotation, net force, gear ratios If you pull backwards on the string, which way will the bike move? Question three When running around a lap of a track, you run at speed v 1 .

Focus on the calculation of:
At what speed do you need to run the second lap, time, distance, and average speed so that the av researchers immersed erage speed of the two laps is equal to 2 times? Question four A normal train is moving along train tracks. Focus on knowledge of: wheels, As the train is moving forwards, what part of the train train tracks is moving backwards relative to the ground?

Page one-survey part one
The students were asked: 'Have you seen this video before?', with the options being yes or no. In part one of the survey, students were prompted to write down their answers to each question in a free-response text box e.g., 'what is your answer to question x (description of question)'. This was followed by a 'next page' button. If any text box was left vacant, the students could not progress. After students moved to page two, they could not go back to page one, so they could not retroactively change their answers.

Page two-solutions video
The solutions video is also from the YouTube channel Veritasium. In it, Derek Muller presents the solutions, completing Dewey's phases by testing the selected hypothesis, see figure 3.
Page two-survey part two Part two of the survey asks: 'after watching the solutions video, did your answer to question one (description of the question) change?' with two options: 'yes I did change my answer' and 'no I did not change my answer'. If the student selects 'I did change my answer' the next statement reads 'I did change my answer because _______'. If the student selects 'I did not change my answer' the next statement reads 'I did not change my answer because _______'. This was repeated for the four questions.

Data and analysis
After removing incomplete responses and those who did not consent, 508 student responses remained. The data were analysed using a mixed-method approach (Wisdom and  Creswell 2013). Analysis of the data from the first part of the survey involved identifying if student answers matched the solution using a 'matching protocol' as outlined in table 2. The 'matching protocol' focuses not on the correctness of student answers but on whether those answers matched the solution provided. The protocol was developed by the first author, who considered the exact solution in the video and the common ways of re-phrasing used by students in their answers. A cornerstone of reflection is the act of comparing and distinguishing one's answer from the given solution, which is why the matching protocol focused on the match between the solution and student answer, rather than 'correctness'. The protocol was discussed with and refined in consultations with the other authors. The analysis from the second part of the survey required coding. One of the researchers immersed themselves in the data, iteratively developing the coding scheme which were discussed with the other researchers. The first group to be coded was where responses did not match and were not modified; referred to as did not match-unmodified. The students' responses were inductively coded, and they aligned with our previous study (Lebedev and Sharma 2019). The coding used in Lebedev and Sharma (2019) had not been pursued from the beginning because the sample was different; public engagement with informal education whose motivations are different to our sample of first year university physics students. The second group, did match-unmodified, was coded deductively, using the earlier coding. The third group, did not match-modified, was inductively coded, as no prior codes have been established for modified responses. The codes for the third group were used for the fourth group, did match-modified.
After all data were coded, a sample of 60 student responses were independently coded by a fourth researcher. The intercoder reliability of 85% affirms the validity and robustness of the codes. Table 3 displays the data corresponding to the first research question. The number of students whose answers matched or did not match are shown, followed by the number who modified or did not modify their answers.

Results and discussion
The numbers of students with matched answers are small for questions one and two, approximately doubling for question four and more than doubling for question three. A substantial percentage of students provided answers that did not match. If we focus on questions one, two and four for not matched answers, most students did modify their answers. However, for not Table 3. The numbers of students whose answers did or did not match for the four questions, and whether they modified or did not modify their answers after watching the solutions video; including percentages in brackets with respect to the total 508 responses. matched answers, the numbers of the students who did not modify their answers despite seeing the solutions video are nontrivial, 24 to 34%. Why they did not modify their answers after watching the solutions video is the subject of research question two. Turning our attention to the pattern in table 3, we discuss the nature of the questions and our matching protocol. We note that students decided to modify/not modify without being aware of the matching protocol or whether their answers were deemed matched/not matched by the researchers. The matching protocol, table 2 was equally discriminating for all questions in that only particular variants on the solution were accepted, and these were applied consistently. Questions one and two draw on integration of concepts and are experiential in nature. Question one is mysterious, inviting a level of informed guessing. Question two was cheeky asking students 'which way will the bike move?' with the solution, 'usually move backwards, but it depends'. For both these questions, students responded, coming up with possible explanations. Most picked up the clues reflecting that their answers did not match the solution, and modified their answers. On the other hand, a nontrivial percentage did not modify their answers even though their answers did not match. We will probe these reflections further through coding the responses. Question three teases one's intuition; it is mathematical in nature, with a clear solution. Those whose answers matched, did not modify; those whose answers did not match, mostly modified. Question four is mischievous, requiring technical knowledge of trains and tracks as well as knowledge of Newton's third law, with a somewhat obscure solution. This question had the largest percentage of students who had answers which were not matched and they did not modify. Despite the fact that the questions were posed as closed questions, all questions were perplexing and one could offer possible explanations as outlined in Dewey's phases.
The second research question explores the types of reflections. Table 4 contain student quotes illustrating the categories of reflection. Note that in our coding, we had started off as inductive and switched to deductive when data saturation had occurred. So the categories and the similarities are a posteriori. Thus, during the act of comparing and distinguishing their answer from the solution, a cornerstone of reflection, students can and do provide similar reasons for both modifying and not modifying answers. It is important to note that individual students were tracked through the four questions; no tendency for consistently providing the same type of reflection for several questions was found. Table 5 presents the number of students in each category for both modified and unmodified when answers were not matched as these are the prevalent categories, see table 3. Next, we discuss the categories drawing on tables 4 and 5.
Categories correct, wrong and repeating the solution, capture answers in which students notice the clues in the solutions and acknowledge that there is agreement or not. A form of similarity, mirrors of each other, is found between correct and wrong-'I did not change my answer because my answer was correct' is very similar (and mirrors) 'I did change my answer because my answer was incorrect'. In the case of repeating the solution, students have actually to the ping pong balls in the liquid understand the explanation but I changed because I was presented with a valid my answer as it was incorrect' argument as to why this explanation 'I had no knowledge of bikes and was superior to my own. This now I do' answer was also supported with evidence that I was able to see for myself.' I am '. . . There really wasn't enough I am 'As I suspected, the answers discontented information for us to be able to find discontented change for various scenarios. the answer, the question was asked There really wasn't enough in a way that made the audience information for us to be able believe there was a set of fixed to find the answer, assumptions that we had to make.' the question was asked in a way that made the audience believe there was a set of fixed assumptions that we had to make.'

Miscellaneous
A student response that Miscellaneous, 'I watched the video could not be coded watched before'A student response that beforehand could not be coded taken action and written the solutions from the video. While this can be seen as a low threshold action, in terms of reflective thinking it can be viewed as a meaning-making process that moves a learner from one experience of the disciplinary content into the next. Combining wrong and repeating the solution, 22 to 32% of students, reconciled their thinking, acknowledging that their answers needed modification. Categories almost correct, almost correct with an explanation or an alternative answer appear in both the modified, and unmodified groups. For example, a typical response is 'I did not change my answer because my answer was almost the same' as compared to 'I did change my answer though my answer was almost the same', both of these responses are coded as almost. Another example would be 'I did not change my answer because my solution would have worked' is very similar to 'I did change my answer; but my answer would also have worked'; both of these responses would be coded as alternative. 'I did change my answers because I was not completely correct and there were two ping pong balls inside too', is categorised as almost correct with explanation, as is, 'I did not change my answer because although I said thick, sticky, heavy substance the ping pong balls added to the mixture change the centre of gravity'. These reflections are mirrors of each other, the students have distinguished their answer from the solution on the video, demonstrating an understanding of the relationships which provides a thread that makes possible the continuity of learning disciplinary content. Table 5 shows that around 15% of the justifications for both modifying and not modifying their answers fell into these categories for question one. For questions two and four, these justifications appeared more for unmodified answers and the justifications were miniscule for question three. The categories discussed thus far fall into the overarching type of reflection captured by Mezirow (1998) as outward focusing where learners are intent on discipline content and tools, reconciling their answers with what is being covered or taught.
The single most popular category of reflections for modifying when answers were not matched is I do not fully understand, 15 to 21% of the students. This is reassuring as these students are able to distinguish that their answer is incongruent with the solution. Sometimes they articulate that they do not know acknowledging gaps in own knowledge, other times that they need to learn more or different things, and yet other times that they have learnt new things. In terms of reflection, they are demonstrating a disciplined way of thinking which requires attitudes that value the personal and intellectual growth of oneself. The category discontented which emerged in both modified and not modified, was not large but should not be neglected. These are responses which express discontent with the question, solution, content or presenter. Given that the questions ranged from somewhat cheeky to inviting a level of informed guessing, the low numbers expressing discontent and falling in this category attests that they rose to the challenge. Again, as students reconcile their thinking, we see the reflections for modified and not modified as mirrors of each other. These categories fall into the overarching type of reflection captured by Mezirow (1998) as inward focusing where learners are intent on their own learning and actions; taking responsibility for their learning or becoming emotionally engaged.
Finally, there were those responses which could not be coded for various reasons and those who had watched the video before.
Note that question 3 again is distinctively different, due to its straightforward mathematical nature, making it the most closed question, that is a question with one direct answer without other possible explanations. Question one is most open, attracting responses such as 'almost correct' to 'alternative solution' for both modified and unmodified. Questions 2 and 4 fall in between. In other words, questions that require integration of concepts and are experiential in nature such as 1 and 2, as well as those which require specific knowledge, such as question 4, are suited for generation of possible explanations in terms of Dewey's phases. Student reflections suggest that these are open questions where multiple plausible answers are offered by students.

Summary
The ubiquity and rise in popularity of online multimedia (Davis 2013) makes it important to research its educational value. It is also worth exploring not just conceptual gains or student engagement (Hill et al 2015, Yeung et al 2016, but reflection, which is under-researched in the online educational video context. This study points to some interesting results with regards to the question, solution & reflection style of videos which takes a typically linear, non-interactive medium and makes it interactive.
For the first research question, we note that the vast majority of students' answers are designated as not matched with the solutions. It is also worth noting that when the students were asked if they changed or did not change their answer, most of the students chose the expected option. That is to say, for those students whose answers did not match, most reported that they modified their answers while for those whose answers did match very few modified their answers. Only very few students whose answers did not match the solutions did not change their answers, self-reporting that they were correct. It seems safe to say that most of the students in this study are able to compare, distinguish and notice the differences between their answers and the solutions provided.
In answering our second research question, as students reconcile their thinking, the reasons provided for not modifying and modifying their answers are similar, or 'mirrors' of each other. The categories themselves demonstrate different types of reflections. Correct, wrong and repeating the solution illustrate a low threshold entry for reflective thinking involving meaning-making of the disciplinary content. Almost correct to alternative answer appearing in both the modified and unmodified groups involves reflective thinking of relationships in disciplinary content such that the learner can offer possible alternate explanations. Do not fully understand for learning is reflective thinking in a disciplined way with an attitude that values the growth of oneself. Discontented can be viewed as emotional engagement. The different types of reflective thinking discussed here, regarded as an effective mechanism to aid learning, are not dissimilar to those reported for adult learners, in particular the overarching patterns of inward focused and outward focused (Mezirow 1998, Hubbs and Brand 2005, Lebedev and Sharma 2019. Different questions in different online physics educational videos serve different purposes (Muller et al 2008, Brualdi Timmins 1998. The questions in our study range from cheeky to inviting a level of informed guessing. Question three, mathematical in nature was perceived by students as a closed question, not inducing different types of reflective thinking. While the other questions were varied, we glean from the categories that they were better aligned with Rodgers' interpretation of Dewey's phases (Rodgers 2002, Dewey 1933. The questions were tricky, somewhat 'cheeky' and mischievous; some were open, some were closed. In our study, such questions induced more reflective activity. Many students rose to the challenge of providing informed guesses or focusing on self learning (Hill et al 2015, Sharma andBewes 2011). All in all, the lesson for physics educators is that they should be tempted to intentionally create contexts for teaching reflective thinking; to provide some cheeky and mischievous formative questions which align with Dewey's phases to engage students in reflective thinking. Despite this task being worth 0.125% of their total grade for the course, the fact that students were prepared to spend time on their responses for this task, is worth noting. In particular, the question, solution & reflection style is rare and it would be worthwhile running similar studies in different contexts and with different topics.