How online learning modules can improve the representational fluency and conceptual understanding of university physics students

The use of online learning resources as core components of university science courses is increasing. Learning resources range from summaries, videos, and simulations, to question banks. Our study set out to develop, implement, and evaluate research-based online learning resources in the form of pre-lecture online learning modules (OLMs). The aim of this paper is to share our experiences with those using, or considering implementing, online learning resources. Our first task was to identify student learning issues in physics to base the learning resources on. One issue with substantial research is conceptual understanding, the other with comparatively less research is scientific representations (graphs, words, equations, and diagrams). We developed learning resources on both these issues and measured their impact. We created weekly OLMs which were delivered to first year physics students at The University of Sydney prior to their first lecture of the week. Students were randomly allocated to either a concepts stream or a representations stream of online modules. The programme was first implemented in 2013 to trial module content, gain experience and process logistical matters and repeated in 2014 with approximately 400 students. Two validated surveys, the Force and Motion Concept Evaluation (FMCE) and the Representational Fluency Survey (RFS) were used as pre-tests and post-tests to measure learning gains while surveys and interviews provided further insights. While both streams of OLMs produced similar positive learning gains on the FMCE, the representations-focussed OLMs produced higher gains on the RFS. Conclusions were triangulated with student responses which indicated that they have recognized the benefit of the OLMs for their learning of physics. Our study shows that carefully designed online resources used as pre-instruction can make a difference in students’ conceptual understanding and representational fluency in physics, as well as make them more aware of their learning processes. In particular, the representations-focussed modules offer more advantages.


Online learning resources
Online learning resources have been used for learning for over 40 years (Harasim 2000). The phrase 'blended learning' is generally used when online learning resources ranging from collaborative activities to assessments are meaningfully integrated into courses with classroom instruction (Black 2002, Ellis et al 2006. With the development of robust technologies and reliable access, most university courses are moving towards some form of blended learning. A popular type of blended learning is pre-lecture online instruction (a form of flipped lecture) which allows for students to be better prepared for lectures (McFarlin 2008, Stelzer et al 2009, Chen et al 2010 and the face-to-face lecture can further adopt the active learning strategies for physics education (Mazur 2009, Georgiou andSharma 2015). Despite an apparent consensus that integrating online learning may be superior (Moreno andMayer 2004, Oncu andCakir 2011), researchers have pointed out the lack of rigorous efforts to demonstrate how such learning can be most effective in post-secondary education (Lack 2013). This opens the opportunity for further research into the uptake of particular designs of online learning, acknowledging that there is considerable ongoing research already in the field. A call along these lines for Australian physics education was made in a national report some ten years ago (Sharma et al 2005).
This paper attempts to share how we designed an online learning resource and how we ascertained its learning effectiveness. The online resource is based on 'blended learning' in that online resource is meaningfully integrated. For this is occur, we had to identify student learning issues in physics to base the learning resources on. Students in science must learn both conceptual information as well as other scientific abilities, one of which is representational fluency, or the use of multiple representations in science (Etkina et al 2006). This study looks at using online resources to teach well researched conceptual understanding, and less researched representational fluency.

Teaching scientific conceptual understanding online
The study of the natural sciences at university requires students to learn a great volume of conceptual information. A typical first year physics course may cover concepts in the areas of mechanics, thermal physics, waves and oscillations, electricity and magnetism, fluids, and quantum physics over just 26 weeks. The volume of information for students to learn, and the increasing diversity of students at university, has put pressure on practitioners to find alternative ways of teaching students science concepts, and using online resources to teach has been a popular solution (Stelzer et al 2009, Chen et al 2010, Stelzer et al 2010, Seery and Donnelly 2012, Lasry et al 2014, Moore 2014. Amongst the literature there are a variety of methods including once-off online exercises, to almost whole courses delivered online. In this paper we draw on one particular example of using online learning to teach concepts before chemistry lectures (Seery and Donnelly 2012). Seery and Donnelly (2012) implemented a series of ten online learning (pre-lecture) resources based on key chemistry concepts to assist first year university students, finding marked improvements in student learning. Their particular style of online learning instruction was effective in teaching key concepts to first year science students. Hence we modelled our concepts stream of online learning modules (OLMs) on this paper to investigate whether there would be similar positive learning gains in physics, and also whether they would impact first year students' representational fluency.
Despite the vast array of research into systematic teaching of science concepts, there have been few attempts to investigate teaching of representational fluency throughout a semester in a university course, and none using weekly OLMs.

Multiple representations and scientific representational fluency
Understanding and using multiple representations is an important skill in the sciences (Aldrich and Sheppard 2000, Roth andBowen 2003) and in particular physics (Beichner 1994, Dufresne et al 1997, Britton 2005, Fredlund et al 2012. Etkina et al (2006) lists this ('the ability to represent physical processes in multiple ways') as the first of seven 'scientific abilities' that must be taught and assessed in introductory university physics (p1). Examples of multiple representations include visual representations (diagrams, maps, and flow charts) and symbolic representations (graphs, equations, and tables) (Gilbert 2004). See figure 1 as an illustration (Redish 2002).
Multiple representations portray relationships where they are not obvious , Goldman 2003 and aid problem solving (Kohl and Finkelstein 2007). Representational Fluency (Hill et al 2014, Nathan et al 2002 describes collectively 'the ability to work within and translate among representations' (p367) (Bieda and Nathan 2009), using representations as experts do Bowen 1999, Kohl andFinkelstein 2005), and learning new representations quickly (diSessa 2004). The mark of a good student in physics is often that they can solve a variety of conceptually challenging problems and this requires fluency in a wide variety of representations to both understand the question and generate an appropriate solution (Dufresne et al 2004).
Making meaning from various representations (semiotics) is conducted differently in various disciplinary discourses. This is often a problem for novice students separating the specialized, technical forms of representations from everyday meanings (Treagust and Chittleborough 2001). This can easily be a barrier to participation in the discipline and Airey and Linder (2009) went so far as to say that fluency in a sufficient variety of specific representations may be a necessity for accessing a disciplines way of knowing. Instructors, and scientific textbooks use much more than the single mode of verbal communication assuming that students have the representational fluency to interpret the information (Lemke 2005). Research indicates that this assumption is not valid as many novice students lack the representational skills and practices of experts or practicing scientists , Woolnough 2000, Rosengrant et al 2009. In particular at The University of Sydney, first year regular physics students appear to lack the representational fluency of more advanced students (Hill et al 2014).
Two questions then arise; first is representational fluency measurable? Second can we create a learning environment which demonstrably fosters the development of representational fluency? The first has been probed through the development and validation of the Representational Fluency Survey (Hill et al 2014) akin to ways in which conceptual learning gains are measured through the Force and Motion Concept Evaluation (FMCE) (Thornton and Sokoloff 1997) or the mechanical wave survey (Tongchai et al 2009). The second question is the focus of this paper.

Teaching scientific representational fluency online
There is some research on instructional methods for improving university students' use of multiple representations (Kohl et al 2007b, Hand andChoi 2010), but a scarcity on improving representational fluency to date to our knowledge. Kohl et al (2007a) investigated whether explicitly teaching and explaining diagrams in physics or using diagrams often and authentically in a semester long programme led to more effective use of diagrams. They found that at the end of semester, both approaches were equally effective with regards to student use of representations.
While we looked to chemistry for literature on teaching concepts online, a discipline with substantial experience teaching representations is mathematics. As a model for our representations-focussed instruction we considered a carefully designed and evaluated Maths Skills programme used at La Trobe University, Melbourne, Australia (Jackson and Johnson 2013). The Maths Skills programme supports the development of mathematical skills amongst university science students as they progress through their semester long courses. Amongst the resources, structured topic worksheets with explicit headings directed students' metacognition towards understanding the purposes and relevance of the material, was found to be effective. We had discussions with this team and used elements of this structure.

Purpose of the study
The purpose was to develop two streams of research based OLMs, a concepts-focussed stream and a representations-focussed stream. Then to investigate which stream would help students improve in two areas, their representational fluency (e.g. using graphs, words or equations), and conceptual understanding (e.g. a knowledge of concepts in mechanics), in order to make recommendations of the best use of OLMs in university physics. Our specific research questions were: (1) How do we develop and implement representations-focussed OLM similar to conceptsfocussed OLM? (2) Can we improve students' learning, (both conceptual understanding and representational fluency) through pre-lecture OLM? (3) Do students recognize the benefit of OLM for physics learning?
The sections below address each in sequence. The rationale aligns particularly with three of the studies discussed earlier. From Seery and Donnelly (2012), we adapted strategies for, 'priming' prior to lectures seeking to enhance understandings. The Concepts OLM emulated this by priming key concepts, while the representations OLM primed key representations. From Kohl et al (2007a) we adapted the 'strongly directed' approach for the representations OLM. This entailed explicitly identifying representations, their affordance and uses. In addition, emphasis was given on requiring students to observe, and enact translations between representations. From Jackson and Johnson (2013) we adapted a specific uniform structure for all the OLM. Each weekly module had three sections consisting of: (1) Information, where content was presented directly to the students, (2) Questions, where internalization of the content was fostered through prompting problems, (3) Reflection, where worth of the content was elaborated with metacognitive questions.

Development
The modules were developed iteratively involving trials with high school students in 2012 and ongoing consultation with lecturers and physics education experts, as shown in figure 2.
A full trial deployment with students at university level occurred in semester 1, 2013. Refinements based on analysis of student responses were made prior to the 2014 deployment. In parallel, targeted consultation through workshops conducted with the wider academic community (Hill and assisted in fine tuning pedagogical aspects. The final collection of representations and Concepts OLM used in semester 1, 2014 was therefore developed using a combination of student responses and expert and practitioner consultation. The results and analysis in this paper focus on the 2014 implementation, see table 1 for a list of topics. There were minimal technical and administrative difficulties, and staff were familiar and comfortable in introducing and referring to OLM in their interactions with students.

Delivery platform
The modules were delivered using Sydney University's eLearning platform 'Blackboard' which allowed for the modules to be completed by students on various devices including mobile tablets seamlessly within their learning management system. Figure

Integrating OLM into regular physics
The implementation occurred within the first year Regular Physics course across a 13 week semester with approximately 850 students. Historically, the course had three one-hour lectures, a one hour workshop tutorial per week, and eight, three-hour experimental laboratory sessions across the 13 weeks. Assessment is via laboratory work, assignments, tutorial participation and a final examination. The course had three modules: mechanics, thermodynamics, and waves and uses Young and Freedman (1996). Into this context we were to move towards flipped-lectures. The first step was to introduce pre-lecture online instruction and demonstrate its effectiveness before changing how the lectures themselves are taught.
Hence there were 12 OLM developed and deployed. They took 15-30 min to complete and could be done in multiple attempts, starting from the second week of the semester. The modules were available from 5pm on Thursday in the previous week and needed to be completed by 10am on Monday which coincided with the first physics lecture of the week. Completion of the modules was worth a nominal 1% of the final mark regardless of correctness of answers.
There were two streams of OLM, each priming work to be covered in the coming week's lectures; the Representations stream comprised 12 modules focussed on representations, the Concepts stream comprised 12 modules focussed on concepts. Each Table 1. Topics for the areas of mechanics, thermal physics, and waves and oscillations. All of the Representsations OLM are highlighted as they all were relevant to the RFS. Five Concepts OLM are highlighted as these were from mechanics therefore relevant to the FMCE. student was randomly assigned to either the Representations or Concepts stream for the semester.
3. RQ 2: Can we improve students' learning (both conceptual understanding and representational fluency) through pre-lecture OLM?
3.1. Measuring the impact of the OLM Students completed pre and post tests which were used for statistical testing and comparing learning gains. To answer research question 1, a conceptual survey, the FMCE (Thornton and Sokoloff 1997) which has been used extensively at the institution was used. To answer research question 2, the, Representational Fluency Survey (RFS) (Hill et al 2014) was used (see appendix). The RFS was developed iteratively including examining validity and reliability, as described in (Hill et al 2014). During the iterative development process student feedback and interviews, along with regular collaboration with an expert panel (including multiple individuals with over 30 years physics education experience), were used to confirm face and content validity. The version of the RFS used in this paper satisfied the criteria for standard statistical tests (difficulty index, point biserial coefficient and Cronbach's alpha). Figure 2 illustrates the study design; structure of the intervention and data collection.

Data collection
Students were randomly assigned to either the Concepts OLM or the Representations OLM. Students who met the following criteria have been included as participants in a particular OLM stream: -completed either the FMCE or the RFS twice, as pre and post test.
-completed more than eight of either Representations or Concepts OLM. Students who met the following criteria have been included as a non-participant in the OLM: -completed either the FMCE or the RFS twice, as pre and post test.
-completed less than four modules. One could argue that the OLM non-participants were more disengaged generally than those in the streams. This is not so. Our data indicate that these students chose to use different learning resources to the OLM and completed either the RFS or the FMCE twice. The nonparticipants persevered in labs, lectures and/or workshop tutorials till the end of the semester. In 2014, there were 406 students who were included in the final analysis, see table 2.

What change in concept test (FMCE) results do the OLMs produce?
We modelled our programme of Concepts OLM on the previous study from Seery and Donnelly (2012) who demosntrated improved conceptual learning. Do our Concepts OLM also produce benefits to conceptual learning? Do our Representations OLM which do not specifically target concepts also have a positive impact on learning concepts? Table 3  provides the mean scores and standard deviations for students from each OLM stream and the non-participants on the FMCE. The distributions were not normal when examined using Kolmogorov-Smirnov test for normality. Using independent-samples Mann-Whitney U tests, we found no statistically significant difference between the distributions of pre-test scores for the Concepts or Representations OLM (p = 0.131). No statistically significant differences were found when comparisons were made with non-participants (independent-samples Mann-Whitney U tests, p = 0.295). Therefore the conceptual understanding, as measured by the FMCE, was the same upon entry. Next we considered improvement across the course of the semester. Using nonparametric related-samples Wilcoxon signed rank test we found a statistically significant increase in scores for both streams and the non-participants (p < 0.001). There was improvement across the course of the semester.
The question then arises, are the improvements of similar magnitudes or does one learning environment offer an advantage? We turn to learning gains, which are a measure of the 'average normalized gain 〈g〉 for a course as the ratio of the actual average gain 〈G〉 to the maximum possible average gain, i.e., 〈g〉 = %〈G〉/%〈G〉 max = (%〈S f 〉-%〈S i 〉)/(100-%〈S i 〉)' (Hake 1998) Table 4 shows that the learning gains on the FMCE are the highest for the Concepts stream, very closely followed by the Representations stream and lowest for the nonparticipants.
In conclusion, these results indicate that both learning modules can improve student performance on a conceptual test. The Representations OLM produce gains almost to the same extent as the Concepts OLM and better than for non-participating students. Our results indicate that well designed representations instruction does facilitate conceptual understandings as it allows particpation in disciplinary discurse (Airey and Linder 2009) required for learning in lectures or any context.  Table 5 provides the mean scores and standard deviations on the RFS for the two streams and non-participants. Student data for the RFS was compared in a similar manner to the FMCE. With the RFS pre test scores, no statistically significant differences were found between the distributions for the two streams and the non-participants. Again, comparing pre and post tests, both streams experienced a significant increase in mean scores (p < 0.001) indicating improvement across the course of the semester.
Unlike the results for the FMCE however, when comparing the post tests, on average, students who were in the Representations OLM stream scored significantly higher on the post Representational Fluency Survey (independent-samples Mann-Whitney U test = 0.011) than those from the Concepts stream.
Considering RFS learning gains, the Representations stream registered the highest gain, followed by the Concepts stream and the non-participants, see table 6. In conclusion, on average both of the OLM streams and the non-participants improved their score on the RFS across semester 1 indicating that the combination of instructional methods did result in improved representational fluency. Both sets of OLM can be seen to be beneficial for learning representational fluency but the Representations stream was most effective. These data show that the representational fluency of university students can be improved through Representations OLM. This successful result is pleasing, but not unexpected as it was targeted through meaningfully integrated blended learning using demonstrated methods such as the strongly directed approach of explicitly teaching representations (Kohl et al 2007a).

Interpreting learning gains
The question now arises, how does this improvement compare with 'normal practice' or other teaching innovations? Here we seek to benchmark learning gains with earlier studies. Learning gains have been graphically represented, on a two-dimensional plot with the x-axis representing the pre-test scores and the y-axis the learning gains, see figure 5. Figure 5(a) is from an extensive study demonstrating that teaching methods employing interactive engagement strategies register higher learning gains than methods employing more traditional approaches (Hake 1998). Figure 5(b), from our institution, illustrates a similar finding, courses with interactive lecture demonstrations (ILDs-where modified predict-observeexplain protocols are intermingled with peer instruction) register higher learning gains than more traditional lectures (non-ILD) (Sharma et al 2010). Consistently studies reveal that particular teaching methods can result in medium gains versus traditional instruction which typically achieves lower gains. Figure 5(c) comprises of learning gains from the FMCE and 5(d) from the RFS from this study.
Noteworthy is that the non-participants registered low gains similar to students with non-ILD instruction as measured by the same test (FMCE) in previous years (comparing 5(b) and 5(c)). This establishes a baseline for our study. Both OLM streams resulted in higher gains for students on the FMCE (medium gains) than non-participants (low gains). In the case of the RFS all three streams fall within the range of medium gain, but again, the students who did complete modules experienced higher gains, with the Representations stream achieving the highest gain.

Data collection and analysis
Student feedback was elicited upon completion of the final Week-13 module, and 12 students who completed the OLM were interviewed at the end of the semester (see sequence in figure 2). Student feedback was in the form of online responses to the following open-ended question (n = 300): 'Name one thing about the OLMs that was helpful for learning physics this semester' Ten face-to-face, semi-structured interviews were conducted individually and one with two students. The interviews sought to probe 'how the modules supported and or hindered learning?' The students were selected using the quota sampling method to ensure representation of the student body. Each interview was 20-40 min in length, participant responses were audio recorded and transcribed by the interviewer to ensure maximum accuracy of both verbal and non-verbal responses.
The analysis of the open-ended responses and interview data occurred after all the data had been collected such that the researchers were immersed in all of the qualitative data while completing the analysis. Iterative coding identified emergent themes which were authenticated by triangulation through different analysis across the two data sources. The interview data provide rich descriptions of the emergent themes. There were three steps in the analysis.
(1) A simple word count of the online responses identified popular words around which the emergent themes could be framed.
(2) Systematic coding of online responses was used to formulate themes.
(3) The themes were validated by an expert and finalized by cross-checking with interview responses. Table 7 presents a word count of the most common words as well as examples of how the students used the words. While the OLM are one learning resource from many (including labs, lectures, tutorials and other online resources), table 7 illustrates that students recognized the strong connection between the OLM and lectures, and in particular improved learning in lectures. The five most common words in conjunction with coding led to three emergent themes with sub-themes, summarized in table 8. Each theme, elaborated below, displays that students do recognize the benefit of completing the OLMs for learning physics.

Results
Theme 1: The students found OLMs prepared them for lectures and other learning.
Various comments from both streams revealed that the students felt that completing the OLMs changed the way they learnt. This was expressed in two ways, some felt that the lectures were easier to understand ('it helps me to understand more and more, much easier to follow the lectures'-Concepts stream, 'by completing the OLM we are not completely clueless in lectures'-Representations stream (theme 1a)) Others felt more prepared for the upcoming material ('It gave me an idea what direction the lectures were heading in'-Concepts stream, 'It made me feel a little more comfortable as I was able to see the 'big ideas' that I would be learning in the following week'-Representations stream (theme 1b)). In essence, these students recognized the purpose of the OLMs to help students 'gain a basic understanding of each topic covered (as an) insight into the materials being covered for each following week' (Concepts stream). The observation of the students that completing the modules helped them 'follow the lectures' (Concepts stream), and 'increase understanding of information during the lecture' (Representations stream) is consistent with pre-lecture priming (Seery and Donnelly 2012). The effectiveness of introducing representations to improve learning in lectures is explained by one student in particular: 'the early introduction to the relevant formulas was extremely helpful as then I was able to relate it to the content and make much more sense of what I was learning' (Representations stream).
Theme 2: The students described the OLMs as directly teaching physics concepts or representations.
Around 19% of respondents commented that the modules improved their 'understanding' of physics (theme 2a). Some students listed particular module topics such as 'thermodynamics' (Concepts stream) or 'drawing ideal gasses' (Representations stream), others spoke more generally about how 'it sometimes explained things better than the lecturer does' (Representations stream) (theme 2a).
As table 7 shows, students from both streams reported that they learnt particular physics concepts from the OLM (theme 2b). ('It was useful in getting the initial idea of the concept which was being explained'-Representations stream). However, only those from the Representations stream (13%) mentioned graphs, equations/formulas or diagrams (theme 2c). Here students recognized that it was an aspect of physics that was being introduced, 'it gave simple hints on reading graphs' (Representations stream). This was the main point of difference between student comments from the two module streams.
Theme 3: The students comment that the OLMs provided other benefits to learning physics. Table 8. The emergent themes and sub-themes from the 300 online responses to the question 'Name one thing about the OLM that were helpful for learning physics this semester'.

Theme
Sub-theme 1a. The OLMs made learning in lectures more effective or efficient 1. OLMs prepared students for lectures and other learning.
1b. The OLMs assisted preparation for the week's learning (lectures, labs and tutorials) 2a. The OLMs explained physics or facilitates learning and understanding 2. OLMs improved student understanding of physics 2b. The OLMs introduced physics content or ideas (concepts) 2c. The OLMs introduced physics graphs, equations or diagrams (representations) 3a. The OLMs encouraged regular physics participation 3. OLMs provided other benefits to learning physic 3b. The OLMs acted as a review (rather than preview) of the week's material The benefits described in this section were not part of the original intention of the OLM but are noteworthy for their potential impact on research into online learning. Students from both streams commented on OLM as a regular activity compelling them to actively participate in physics, impacting positively on their learning experience (theme 3a). ('They made sure I did some physics work every week'-Concepts stream, 'The compulsory evaluation of our learning each week was very helpful'-Representations stream). Some students requested that OLM be given for post-lecture revision (theme 3b) ('rather than the online modules trying to prepare for the lectures, they felt like a more appropriate and encouraging reminder of the things mentioned in the lecture instead'-Concepts stream). These students valued the OLM as a metacognitive reflective tool ('it forced me to do a quick mental summary of things I had learnt that week'-Concepts stream) and as an 'evaluation of our learning each week' (Representations stream). Given the numbers of students who value the OLM for pre-lecture priming, whether to make them available afterwards, or for longer time periods is a challenging decision to make for educators.

Interview responses.
There was a greater difference between the streams in the interviews than in the online responses, however, the responses matched the themes identified in table 8. Students who completed the Concepts stream recognized the benefit of the almost flipped lecture approach.
Initially for the first two or three weeks I thought they were pointless… but by doing the modules we do have a rough idea of what we are going to learn so when it ends up in lectures we know what the lecturer is telling us so we do not have to stop or pause it and ask him for every single time rather we can just move on with the class. (Concepts stream) (theme 1a).
They also believed that the modules played a role in priming prior knowledge, to optimize learning in lectures.
Obviously you can not show up to a lecture and understand 100% what they are saying without some prior knowledge, so I feel that the online stuff did give me that prior knowledge that you needed. (Concepts stream) (themes 1a and 2a).
Some students from the Representations stream also recognized a shift in their 'subconscious' attitudes towards lectures and were able to describe the metacognitive shifts that the modules facilitated. at a subconscious level it is working so you could maybe look at the lecture in new ways. (Representations stream) (theme 1).
Furthermore, students recognized how priming explicit representations freed up cognitive space so more complex ideas could be understood in lectures.
(The modules) told me about the graph and how it works… when they started talking about how it is to be applied and what it means, as opposed to being stuck with how it works and being behind, I already knew. (Representations stream) (themes 1a and 2c).
Analysis of both the online responses and participant interviews illustrate how students were positive towards the OLM regardless of the focus on representations or concepts. In addition, they recognized that they were not stand-alone, but assisted learning in lectures (theme 1). In the case of the Representations stream, it allowed for a particular barrier to learning to be lowered (theme 2c) which supports the quantitative findings that the Representations stream have the highest learning gains according to the representational (RFS) measure, and almost equally high learning gains as the Concepts stream according to the conceptual (FMCE) measure.

Online resource development processes
There were two notable factors in the success of the OLM at The University of Sydney. First was the research-based design; drawing on previous studies increased the likelihood of success of our move towards blended learning. We recommend that educators investigating blended learning consider the literature in (but not limited to) this paper and where possible, consult authors and educators attempting similar strategies.
The second factor was undertaking trials as shown in figure 3. The process of trailing physical worksheets in two high schools resulted in substantial changes which ensured that the modules were communicating what they were designed to communicate. The ideal would be to trial online modules on a small scale, but technological constraints prevented this. Hence, the first full implementation in 2013 is viewed as another trial. This study reports results from the 2014 deployment of the OLM as we consider this to be 'going live'. The three year investment has resulted in an online learning resource that will need minimal, if any tweaks in the near future assuming that the syllabus is not altered. And we have evidence that the resource improves student learning and engagement. We see our study as an opportunity to analyze results and understand student learning and use of online resources even further. We recommend that educators consider trials prior to full deployment of learning resources.

Deployment and management strategies
Reflecting on student participation and students' comments (see research question 3), there are a number of lessons that can be learnt from our particular implementation of OLM in first year physics.
(a) Offering 1% for completing 11 out of 12 OLM had some consequences: (i) a 1% incentive was sufficient to get most students completing the weekly online activities. (ii) The OLM were awarded marks for participation rather than correct answers. This encouraged authentic participation and for students to take responsibility for their own learning. Students did not take advantage of the system. (iii) Clear communication is necessary as some students thought that by missing two OLM they were no longer able to attain any marks. The concept of pro rata marks needs to be stressed. (iv) When marks are associated with any activity some students will seek clarification that marks have been awarded. A system needs to be in place to regularly monitor the online system and student emails. It is important to support students with access and completion issues as technical glitches can occur.
(b) Communication of the purpose of the pre-lecture OLM was important for encouraging participation and managing student expectations: (i) Students were informed that the primary purpose of the OLM was to prepare for lectures. This was recognized by students as a helpful element of the OLM. (ii) Students were reminded at the start of each new physics topic with a different lecturer (Thermal Physics, Oscillations and Waves) that the OLM would continue for these topics. (iii) Students were not told that the OLM were an 'innovation' in this course. From the student perspective, the OLM (and blended learning) were simply a normal part of the course.
(c) Making OLM available from 5pm Thursday until 10am Monday morning was appropriate for pre-lecture online activities: (i) An average of 15 min (student times typically ranged from 8-30 min) was appropriate. We saw significant changes in student learning. There was appreciation of the OLM rather than complaints. (ii) Thursday until Monday morning gave the students enough flexibility but recognized that many students would complete the OLM at the last minute. Therefore giving a larger time window for students would be unnecessary. (iii) Having access available over the weekend was necessary (as many students did the exercise on the Sunday) but also required periodic monitoring over the weekend to troubleshoot problems that inevitably arose.

5.
3. Which is more beneficial for pre-lecture OLM: introducing physics representations or physics concepts?
The results of research question 2 showed clearly that both streams of OLM were beneficial for student learning. Therefore we would encourage any tertiary science educator who is using completely classroom-based instruction to consider blended learning of pre-lecture instruction with either representations or concepts. Both of the streams produced similar learning gains (Representations: 0.198, Concepts: 0.219) on the concept survey (FMCE), while those who elected not to complete the OLM but did participate in the course registered gains in the low range (0.147). Despite this, for students to develop scientific representational fluency, the representations-focussed OLM were clearly more effective on the RFS (Representations: 0.329, Concepts: 0.265). In comparison, the gain for those who did not complete the OLM, was 0.242 is similar to the gain for the concepts stream. Therefore the results of this investigation would suggest that introducing representations through OLM is the better pre-lecture instruction option for student cohorts like the first year Regular physics students at The University of Sydney.
It is hoped that this result and implication can be used by other scientific disciplines too as while representational fluency here is nuanced for physics students, it is an interdisciplinary concept (Hill et al 2014). Therefore educators in chemistry, biology, and environmental sciences could consider the representations that are taught and how they can best introduce them through blended learning or otherwise.
It could be suggested that the ideal instruction incorporates both concept-focussed and representation-focussed teaching. We would agree and argue that explicit representationfocussed instruction is often lacking in many scientific education settings. However, in the case of pre-lecture activities, where there is limited time in preparing for further teaching in lectures, this study demonstrates that representation-focussed instruction should be prioritized.

Conclusion
The implementation of online pre-lecture learning modules in a first-year university calculusbased physics course resulted in improved learning gains on both conceptual and representational reasoning tests. Completing these modules, in addition to regular course instruction, increased student conceptual understanding and representational fluency greater than regular course instruction alone. Results over two years indicate that student representational fluency can be developed through targeted teaching strategies in particular explicitly introducing students to physics representations weekly throughout the semester. Furthermore, qualitative analysis supports the quantitative data and also shows that the students themselves recognize both intended and unintended benefits of OLM.

Appendix. The representational fluency survey
The full seven item survey can be found at http://physics.usyd.edu.au/super/RFS/The% 20Representational%20Fluency%20Survey.pdf In the research described in this paper, item 6 was excluded from analysis as recommended by the creators of the survey (Hill et al 2014), resulting in a maximum mark of 18 on the RFS.