INTRODUCTION

Spurred by scientific advances of the last century and the accelerated rate of innovations (Froehlich & Messmann, 2017), technology has reached an unquestionably important status in our daily personal and professional lives. Indeed, we do live in a world permeated by an abundance of digital technologies. Universities, too, more and more become reflective about the challenges and opportunities that this transition into what can be called the “digital age” imparts. However, this digitization is not always reflected in the learning environments of institutions of higher education; often necessary technological resources are not made available for the students. This raises questions about the adequacy of the instructional outcomes; after all, the learning context should be related to the context of application, as suggested by especially the transfer of training literature (Baldwin & Ford, 1988; Froehlich & Gegenfurtner, Forthcoming).

Flipped or inverted learning as a “type of technology-enhanced pedagogy” (Lo, Lie, & Hew, 2018, p. 150) has recently received a lot of attention in diverse domains such as humanities (Kong, 2014), nursing (Njie-Carr et al., 2017), statistics (Boevé et al., 2017; Peterson, 2016), mathematics (Tawfik & Lilly, 2015), languages (Chuang, Weng, & Chen, 2018), social work (Holmes, Tracy, Painter, Oestreich, & Park, 2015), or education (Al-Zahrani, 2015). Here, flipped classroom describes the instructional strategy where content is delivered before the class is taught (Reidsema, Kavanagh, Hadgraft, & Smith, 2017; Talbert & Bergmann, 2017). This is often done via videos—which I also refer to as “flipped elements” in this text. The effectiveness of flipped classrooms has been studied with respect to a host of different outcomes such as critical thinking skills (Kong, 2014), course satisfaction (Peterson, 2016), self-regulation (Sun, Wu, & Lee, 2017), creative thinking (Al-Zahrani, 2015), or how this approach lends support to other teaching activities such as problem-based learning (Tawfik & Lilly, 2015). In sum, previous research suggests many positive outcomes stemming from flipped classroom elements.

Boevé et al. (2017) examined students’ study behavior during a course using flipped instruction. However, contrary to many other findings in the literature, they did not find a substantial difference in students’ study behavior when contrasting regular and flipped classes. They highlight that more understanding is needed, when the flipped classroom pedagogy may be most fruitfully applied. It is this gap I seek to slightly narrow with this article. Specifically, while the flipped classroom pedagogy itself is mostly interpreted to be technology-based, research on using this methodology specifically in learning environments that are lacking technological features is scarce. In other words, we do not know much about how different leaning environments in terms of technological resourcefulness and the efficiency of flipped classroom elements are related to each other. In the context of this article, I focus on rather basic technological features, such as the availability of computers for the students in an applied statistics classroom.

I propose that flipped elements may function as a buffering mechanism in such a way that any differences between learning environments rich in technological resources and learning environments poor in technological resources are decreased. This, in turn, may also curb differences in terms of student outcomes such as student learning or student performance. In more practical terms, this means that flipped elements may come to the aid of learning environments that do not possess technical resources—which, in the case of the empirical demonstration below, means the hardware and software to perform applied statistics. The rationale for this proposition is that flipped elements transfer the demand for technology (i.e., having access to a computer with all the needed software for a specific course) from the teaching environment (e.g. a lab or a seminar room) to the student (e.g. privately owned computers or shared facilities at the university). If this proposition holds true, this has important implications for how universities (especially those that could be considered poorly equipped in terms of technology) could master the transition into the digital age in a sustainable and financially viable manner.

In order to generate primary data to test the proposition outlined above, I executed a design-based research project (Anderson & Shattuck, 2012; Euler & Sloane, 2014). Following the nature of design-based research projects (which will be described in more detail in the Material and Methods section), this investigation is embedded in my own teaching practice. The contribution that this article aims to make, therefore, cannot be a robust and generalizable test of this proposition. However, I do aim to test the general viability of the idea so that further research can investigate the proposition in greater detail, using more robust methods, and in a more informed and efficient way. Also, I do discuss the implications for higher education teachers and the universities they are teaching in.

The rest of this paper is structured as follows. First, I describe the methods and all the data sources tapped in the empirical part of this study. I briefly explain the nature of design-based research projects as well as the background in which this particular project is embedded in. Next, I provide information about the intervention that was implemented and the data sources that were tapped in order to measure the effect of the intervention. I briefly present the results and interpret and discuss them in light of universities and the opportunities and challenges the digital age brings for them.

MATERIAL AND METHODS

In this section, I define and present the process of design-based research and provide the necessary background information about the project. Also, I describe the flipped classroom intervention that is in the focus of this article. Last, I explain how evaluative data was collected and analyzed.

Overview of the method

In the following pages, I report a design-based research project (Choi & Lee, 2008) to provide some preliminary empirical evidence for the previously presented proposition and the rationale behind it. Design-based research is often executed as a series of three major steps (Anderson & Shattuck, 2012): First, the research is situated in a real teaching context, where a potential point for improvement was identified. Second, within this context, an intervention is implemented that is estimated to be powerful enough to have a notable effect. Third, qualitative and quantitative data are purposefully integrated (Johnson & Onwuegbuzie, 2004; Schoonenboom, Johnson, & Froehlich, 2018) to assess the intervention. This then is usually repeated so that the proposed intervention can be refined iteratively (see Figure 1). In this particular study, we deviate from the latter, as I do not make a comparison over time (e.g. having pre- and post-intervention evaluations of one course) but over courses that were taught in parallel. Specifically, in this particular design-based research project, the overall research strategy is one of making comparisons between two courses in very different contexts and learning environments, which are described in the next section.

Background

I have chosen two courses as a basis for comparison. The chosen courses—which I from now on will refer to as “A” and “B”—lend themselves for this comparison as they are homogeneous in defining features of the course; for instance, they share the same goals (prepare students to execute a quantitative study on their own for their thesis work, decrease statistic anxiety, increase statistical self-efficacy) and the intended audience (first year Bachelor students). They are also similar in terms of the broader institutional context: in both cases, public universities located in Austria.

However, the courses are very different when it comes to the actual learning environment. In setting A, I teach the course to 25 students in a computer lab; every student has access to appropriate hardware and also commercial statistics software (also outside the course). In setting B, I teach 150-160 students in an auditorium without any technical equipment (except for a projector) that does not provide enough space for students to work on their own laptops. Table 1 gives an overview of the differences.

Table 1. Overview of the different settings

Setting	A	B
Institution	University of applied science	University
Level	Bachelor	Bachelor, but also Master/PhD students present
Class size	25	151
Room	computer lab	lecture room
Prior knowledge	homogeneous	heterogeneous

This project was originally part of a larger data collection effort that comprised more settings, including some that did not use flipped classroom elements. However, given that the differences in outcomes is not the goal of this particular paper and since these differences are already very well documented in the literature, the respective courses are ignored to increase clarity of presentation.

Intervention

The intervention in this design-based research study is the introduction of flipped-classroom elements (videos in the format of “slidecasts” about statistical theory—e.g. the central limit theorem or confidence intervals—and “screencasts” for more application-oriented videos— e.g. how to perform a specific statistical test and how to interpret it). The videos were produced by myself specifically for the courses to be studied. The total duration is about five hours, which amounts to roughly 25% of the total teaching time in the classroom. I deem this high percentage vital in order to make the intervention an important part of the courses, which increases the likelihood that there is a measurable impact.

The intervention was planned and executed in the same manner in both courses. Specifically, each flipped element was integrated in a teaching strategy of three steps:

1. Before each lecture, the students received videos to watch that introduced the topic of the lecture theoretically and showcased an exercise directly in a statistical software package. The students were not required to have any prior knowledge about the content being taught.
2. Then, the students were required to solve an exercise of similar difficulty using the same statistical software package. In setting A, the courses were given as extra resources with the explicit recommendation to watch them before each lecture in the classroom. In setting B, viewing the videos before the lecture was mandatory and tested using a graded quiz (cf. Taylor, 2015). While these different approaches are interesting in their own right, they are not the focus of this particular paper.
3. During the lecture in class, I answered any questions on the theoretical video and showed a step-by-step solution to the assignment(s). Afterwards, a more complex exercise about the same topic was done directly in class. While the students were able to do the exercise directly in class in the computer room, this was not feasible in setting B, where this part was less interactive and rather teacher-centered instruction.

Instruments

I compare the courses based on several both quantitative and qualitative criteria:

In terms of quantitative criteria, I draw from three major sources: First, a quiz that was a formal part of both courses. In both settings, the quiz served as a preparatory exam before the real exam at the end of the course. This preparatory exam, however, was taken quite seriously and completed by all students. Second, I disseminated surveys that contained psychographic questions about statistics anxiety (Onwuegbuzie & Wilson, 2000) (23 items, sample item: “How anxious do you feel when you need to interpret a statistic in a journal article?”; answers from 1 = “no anxiety” to 5 = “strong feeling of anxiety”) and statistical self-efficacy (Finney & Schraw, 2003; Larwin, 2014) (seven items, sample item: “How confident do you feel in selecting an appropriate statistical test for a given research question?”; answers from 1 = “not confident at all” to 5 = “totally confident”), as well as standardized course- and teacher-evaluation scales. The questionnaire that contained the psychographic information as well as the standardized evaluation was completed by 23 and 22 students in settings A and B, respectively. Third, the quantitative parts of the respective course evaluations are used.

In terms of qualitative criteria, I rely on field notes and reflection memos (Saldana, 2009) made throughout the whole teaching process as well as the information students provided at various evaluation points throughout and at the end of the course. Additionally, both courses applied for teaching awards at their respective institutions and, hence, received thorough review by a jury of senior higher education teachers.

Analytical strategy

For the analysis, it needs to be acknowledged that all the data collected describes a single case. The analytical aim, therefore, cannot be to deliver generalizable results. Instead, I test whether preliminary evidence in support of the proposition can be generated. Such first evidence may form an important basis for more generalizable (but also more costly) research (cf. Froehlich, Liu, & Van der Heijden, 2018).

Each data source presented can hardly be considered robust evidence on its own. However, the multitude of different perspectives associated with the different data sources allows many opportunities for triangulating the evidence and to form an overall conclusion (cf. Jick, 1979). We interpret each of the following comparisons as in support of the proposition that the introduction of flipped elements can help level the playing field between learning environments rich in technological resources and those not so, if the absence of negative differences between setting A and B is indicated. For comparison of quantitative data, t-tests are used.

RESULTS

In this section, I present the results of the comparison of all data collected.

Statistical self-efficacy was, on average, rated to be quite low in both courses, but the difference between setting A (M = 2.4, SD = 0.7) and setting B (M = 3.2, SD = 1.0) was statistically significant (T = -3.1, p ≤ 0.01). This indicates some support to the proposition (especially because the setting rather poorly equipped scored higher regarding this measure).

For statistic anxiety, the means of setting A (M = 2.6, SD = 0.7) and setting B (M = 2.2, SD = 0.8) do not show any statistically significant differences between the learning environments. This lends weak support to the proposition.

As concerns the evaluative scales of teaching performance (A: M = 4.5, SD = 1.1; B: M = 5.4, SD = 1.3) and the overall course quality (A: M = 4.8, SD = 1.0; B: M = 5.6, SD = 1.3), statistically significant differences (p ≤ 0.05) exist in support of the proposition (again, especially because the setting rather poorly equipped scored higher regarding these measures).

The qualitative data are somewhat more difficult to weigh against each other given the contextual differences, but they do not suggest a stark difference in any direction between the two courses. Students in both courses highlighted the videos as the most importance resource for preparing for each class and the exam. The independent juries of the applied for teaching awards, too, gave very positive evaluations of both courses. The course in setting B won a university-internal teaching award, the course in setting A was nominated for a nation-wide award by the institution. Both applications were highly competitive processes.

Table 2 depicts all these sources of evidence together with the results and their interpretation. As stipulated before, each single base of evidence can hardly be considered robust evidence in terms of testing the proposition. However, it is notable that all the results point into the same direction, which does give some confidence in stating that the proposition is supported by the mixed data presented.

Table 2. Overview of the research results

Data	A	B	Analysis and Interpretation	Conclusion
Psychographic measurement of statistical self-efficacy	M = 2.4, SD = 0.7	M = 3.2, 1.0	Statistically significant difference; setting B scores higher	Supports the proposition
Psychographic measurement of statistic anxiety	M = 2.6, SD = 0.7	M = 2.2, SD = 0.8	No statistically significant difference	Weakly supports the proposition
Evaluative scales of teaching performance	M = 4.5, SD = 1.1	M = 5.4, SD = 1.3	Statistically significant difference; setting B scores higher	Supports the proposition
Evaluative scales of overall course quality	M = 4.8, SD = 1.0	M = 5.6, SD = 1.3	Statistically significant difference; setting B scores higher	Supports the proposition
Student qualitative assessment	highly rated	highly rated	No important difference found	Weakly supports the proposition
Jury qualitative assessment	highly rated	highly rated	No important difference found	Weakly supports the proposition

DISCUSSION

In this article, I argue that flipped-classroom elements (Reidsema et al., 2017; Talbert & Bergmann, 2017) may help higher education teachers in learning environments poor in technological resources to deliver technology-based courses. Specifically, I proposed that flipped elements may help to curb negative student outcomes that are a direct result of learning environments poor in technological resources. As a rationale for this proposition I suggested that flipped elements transfer the demand for technology from the teaching environment to the student. In order to generate first evidence for this proposition, I have implemented a design-based research project (Choi & Lee, 2008) that features an intervention (the introduction of flipped elements) that was implemented in two very different contexts. Also, different sources of qualitative and quantitative data were tapped to procure information about the efficiency of that intervention. All data generated points into the same direction: The course in the less technologized learning environment of setting B is not evaluated to be more negative in any way than the more technologized learning environment of setting A. I interpret this to be in support of the proposition.

Limitations and implications for further research

The study presented here was not intended to produce fully conclusive results. However, the aim was to generate early-stage evidence that gives some more security when venturing into more elaborated research designs to investigate this topic. Arguably, the most important limitation concerns the question about generalizability. All the data were used to contrast two single cases; given that a plethora of factors were not part of this research (most prominently, the teacher), no generalization is possible. Still, I did use a host of very different indicators to test the proposition and they all point into the same direction. Also, although not the primary goal of this research project, the findings are in line with the positive outcomes of flipped classroom designs found elsewhere in the literature (e.g. Peterson, 2016; Sun et al., 2017). This gives some confidence in the robustness of the results generated in this article.

This study hopes to facilitate the conduct of further investigations that may help to bridge this gap and that are able to produce more generalizable outcomes.

Implications for practice

To the best of my knowledge, the mechanism of buffering potentially negative effects of learning environments poor in technological resources has neither been proposed nor tested before. These first results, however, indeed suggest that flipped elements may help to alleviate this issue. This has a few important implications for practice.

For universities, this buffering effect may be an important ingredient during the transition into the digital age. This is especially true for larger institutions (such as setting B), where the financial burden of providing the technological resources for the students may put a lot of strain on the financial balance and the flexibility as an institution. If, however, flipped elements may already be enough to limit negative effects as suggested by the proposition, this may not be necessary.

For higher education teachers, the buffering effect described here may represent another advantage of the flipped-classroom pedagogy to be added to the list of commonly defined advantages (e.g. Taylor, 2015, or see the list of example outcomes given in the introduction). This may also increase the motivation to subscribe to this model of teaching. Since the described effect requires an investment on the side of the students—they need access to a computer with the required software—, it is important to recognize the requirements when planning the course and the flipped elements. Specifically, it is recommended to only use software that is either freely available or that is accessible via the shared workstations at the university. While teaching in an environment where students have different computer setups, operating systems, etc. definitely becomes more complex and challenging for the teacher, it also furthers technological competencies on the side of the students more than a “ready-to-use” computer lab at the university does. On the side of the teacher’s input, it is important to highlight the workload associated with creating flipped elements (see for instance Taylor, 2015). This stresses the need for finding more sustainable ways of creating and sharing open educational resources (Caswell, Henson, Jensen, & Wiley, 2008; Downes, 2007), as universities aim to leverage the opportunities and master the challenges of the digital age.

REFERENCES

1. Al-Zahrani, A. M. (2015). From passive to active: The impact of the flipped classroom through social learning platforms on higher education students’ creative thinking: From passive to active. British Journal of Educational Technology, 46(6), 1133–1148. doi:10.1111/bjet.12353
2. Anderson, T. & Shattuck, J. (2012). Design-Based Research: A Decade of Progress in Education Research? Educational Researcher, 41(1), 16–25. doi:10.3102/0013189X11428813
3. Baldwin, T. T. & Ford, J. K. (1988). Transfer of training: A revire and directions for future research. Personnel Psychology, 41(1), 63–105. doi:10.1111/j.1744-6570.1988.tb00632.x
4. Boevé, A. J., Meijer, R. R., Bosker, R. J., Vugteveen, J., Hoekstra, R., & Albers, C. J. (2017). Implementing the flipped classroom: an exploration of study behaviour and student performance. Higher Education, 74(6), 1015–1032. doi:10.1007/s10734-016-0104-y
5. Caswell, T., Henson, S., Jensen, M., & Wiley, D. (2008). Open Content and Open Educational Resources: Enabling universal education. The International Review of Research in Open and Distributed Learning, 9(1). doi:10.19173/irrodl.v9i1.469
6. Choi, I., & Lee, K. (2008). Designing and implementing a case-based learning environment for enhancing ill-structured problem solving: classroom management problems for prospective teachers. Educational Technology Research and Development, 57(1), 99–129. doi:10.1007/s11423-008-9089-2
7. Chuang, H.-H., Weng, C.-Y., & Chen, C.-H. (2018). Which students benefit most from a flipped classroom approach to language learning?: Flipped classroom does not fit all students. British Journal of Educational Technology, 49(1), 56–68. doi:10.1111/bjet.12530
8. Downes, S. (2007). Models for Sustainable Open Educational Resources. Interdisciplinary Journal of E-Learning and Learning Objects, 3(1), 29–44.
9. Euler, D., & Sloane, P. F. E. (Eds.). (2014). Design-based research. Stuttgart: Franz Steiner Verlag.
10. Finney, S. J., & Schraw, G. (2003). Self-efficacy beliefs in college statistics courses. Contemporary Educational Psychology, 28(2), 161–186. doi:10.1016/S0361-476X(02)00015-2
11. Froehlich, D. E., & Gegenfurtner, A. (Forthcoming). Social support in transitioning from training to the workplace: A social network perspective. In H. Fasching (Ed.), Beziehungen in pädagogischen Arbeitsfeldern [Relations in pedagogical work]. Bad Heilbrunn: Klinkhardt.
12. Froehlich, D. E., Liu, M., & Van der Heijden, B. I. J. M. (2018). Work in Progress: The Progression of Competence-Based Employability. Career Development International.
13. Froehlich, D. E., & Messmann, G. (2017). The Social Side of Innovative Work Behavior: Determinants of Social Interaction during Organizational Innovation Processes. Business Creativity and the Creative Economy, 3(1), 31–41.
14. Holmes, M. R., Tracy, E. M., Painter, L. L., Oestreich, T., & Park, H. (2015). Moving from Flipcharts to the Flipped Classroom: Using Technology Driven Teaching Methods to Promote Active Learning in Foundation and Advanced Masters Social Work Courses. Clinical Social Work Journal, 43(2), 215–224. doi:10.1007/s10615-015-0521-x
15. Jick, T. D. (1979). Mixing Qualitative and Quantitative Methods: Triangulation in Action. Administrative Science Quarterly, 24(4), 602–611.
16. Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed Methods Research: A Research Paradigm Whose Time Has Come. Educational Researcher, 33(7), 14–26.
17. Kong, S. C. (2014). Developing information literacy and critical thinking skills through domain knowledge learning in digital classrooms: An experience of practicing flipped classroom strategy. Computers & Education, 78, 160–173. doi:10.1016/j.compedu.2014.05.009
18. Larwin, K. (2014). Statistics Related Self-Efficacy A Confirmatory Factor Analysis Demonstrating a Significant Link to Prior Mathematics Experiences for Graduate Level Students. Mathematics Education Trends and Research, 2014. Retrieved from http://www.ispacs.com/journals/metr/2014/metr-00022/abstract/
19. Lo, C. K., Lie, C. W., & Hew, K. F. (2018). Applying “First Principles of Instruction” as a design theory of the flipped classroom: Findings from a collective study of four secondary school subjects. Computers & Education, 118, 150–165. doi:10.1016/j.compedu.2017.12.003
20. Njie-Carr, V. P. S., Ludeman, E., Lee, M. C., Dordunoo, D., Trocky, N. M., & Jenkins, L. S. (2017). An Integrative Review of Flipped Classroom Teaching Models in Nursing Education. Journal of Professional Nursing, 33(2), 133–144. doi:10.1016/j.profnurs.2016.07.001
21. Onwuegbuzie, A. J., & Wilson, V. A. (2000, May). Statistics Anxiety: Nature, Etiology, Antecedents, Effects, and Treatments: A Comprehensive Review of the Literature. Paper presented at the Annual meeting of the Mid-South Educational Research Association, Lexington, Kentucky. Retrieved from https://files.eric.ed.gov/fulltext/ED448202.pdf
22. Peterson, D. J. (2016). The Flipped Classroom Improves Student Achievement and Course Satisfaction in a Statistics Course: A Quasi-Experimental Study. Teaching of Psychology, 43(1), 10–15. doi:10.1177/0098628315620063
23. Reidsema, C., Kavanagh, L., Hadgraft, R., & Smith, N. (Eds.). (2017). The Flipped Classroom: Practice and Practices in Higher Education (1st ed. 2017 edition). New York, NY: Springer.
24. Saldana, J. (2009). The Coding Manual for Qualitative Researchers. Los Angeles, CA: Sage.
25. Schoonenboom, J., Johnson, R. B., & Froehlich, D. E. (2018). Combining Multiple Purposes of Mixing Within a Mixed Methods Research Design. International Journal of Multiple Research Approaches.
26. Sun, J. C.-Y., Wu, Y.-T., & Lee, W.-I. (2017). The effect of the flipped classroom approach to OpenCourseWare instruction on students’ self-regulation: Flipped classroom approach and OpenCourseWare. British Journal of Educational Technology, 48(3), 713–729. doi:10.1111/bjet.12444
27. Talbert, R., & Bergmann, J. (2017). Flipped Learning: A Guide for Higher Education Faculty. Sterling, Virginia: Stylus Publishing.
28. Tawfik, A. A., & Lilly, C. (2015). Using a Flipped Classroom Approach to Support Problem-Based Learning. Technology, Knowledge and Learning, 20(3), 299–315. doi:10.1007/s10758-015-9262-8
29. Taylor, A. (2015). Flipping Great or Flipping Useless? A review of the flipped classroom experiment at Coventry University London Campus. Journal of Pedagogic Development, 5(3). Retrieved from https://journals.beds.ac.uk/ojs/index.php/jpd/article/view/230