Effective Pedagogical Strategies for STEM Education from Instructors’ Perspective: OER for Educators

The Massachusetts Institute of Technology (MIT) OpenCourseWare (OCW) was launched in 2001. It is one of the earliest Open Educational Resources (OER). MIT OCW has published more than 2,400 courses which are available at no cost, the majority of which are STEM related. The purpose of this exploratory study was to examine the pedagogical strategies through reviewing instructor insights of 15 MIT OCW STEM courses using thematic analysis. The most effective pedagogical strategies used found by instructors were active learning, personalizing instruction, engaging learners, providing feedback, building learning community, and clarifying learning objective. Instructors used in-class formative assessment, such as quizzes and oral exams, for just-in-time teaching and online automatic assessment environments for students’ selfassessment. The primary summative assessments were final exams and projects. Instructors encountered challenges such as assessing students’ learning and changing pedagogical beliefs. Implications for practice were discussed as well.


Introduction
One of the earliest Open Educational Resources (OER) is the Massachusetts Institute of Technology (MIT) OpenCourseWare (OCW) (Bonk, 2009;MIT News, 2001), which was launched in 2001. It was intended to share MIT course materials with the public for free. As of August 2019, more than 2,400 courses were available, with approximately 170 million visitors to their website (MIT OpenCourseWare, 2019b). Nine percent self-reported as educators (MIT OpenCourseWare, 2019a). The majority of the courses are related to science, technology, engineering, and mathematics (STEM) subjects such as Electrical Engineering (EE) and Computer Science (CS).
Open Educational Resources (OER) was previously examined regarding design, development, disseminating methods, and quality by Kimmons (2015), cost-saving by Wiley, Hilton, Ellington, and Hall (2012), and impact by d 'Oliveira, Carson, James and Lazarus (2010). However, STEM pedagogical strategies were not yet considered. Thus, the purpose of this study is to examine MIT OCW STEM instructor insights in order to inform other STEM educators regarding effective pedagogical strategies and assessment methods and possible challenges.
The following three research questions guided this study: 1. What are the effective pedagogical strategies for MIT OCW EE and CS courses? 2. What are the commonly used assessment methods of MIT OCW EE and CS courses? 3. What challenges do the instructors perceive they have while teaching MIT OCW EE and CS courses?

STEM Education
The National Science Foundation (NSF) and other related professional societies have focused on the basic and applied research in STEM and to improve the quality of STEM education (Fairweather, 2008). The attention to STEM was a response to the decrease of the number of students selecting STEM majors and the needs of STEM related employees (Center for Science Mathematics and Engineering Education, Committee on Undergraduate Science Education, 1999;National Science Foundation, 1996). The Committee of Science, Technology, Engineering, and Math Education (CoSTEM), a branch of the National Science and Technology Council, was formed to reform STEM education from K-12 to the higher education levels in order to build a pipeline of jobs for the development of the economy (CoSTEM, 2013). Low quality college teaching in STEM courses is a critical issue in higher education (Seymour & Hewett, 1997), promoting the search for effective pedagogical practices. This includes instructors' professional development programs (Wulff & Austin, 2004) for teaching innovation. Fisher, Zeligman, and Fairweather (2005) indicated that pedagogical reforms and innovations in engineering courses significantly enhanced student learning outcomes, including ill-structured problem-solving skills.

Active Learning
Active learning is potentially more effective than the traditional teacher-centered instruction in terms of enhancing student learning and increasing student retention in STEM education (Freeman et al., 2014;Lund & Stains, 2015;Michael, 2006;Prince, 2004). Active learning could address student learning needs and promote critical thinking (Kim, Sharma, Land, & Furlong, 2012). For example, Freeman et al. (2014) conducted meta-analysis on the effectiveness of teaching in traditional lecturing vs. active learning approach in STEM courses, and the latter was frequently accompanied by increased student performance.
Blended learning is another active learning strategy in which both online and face-to-face instruction or learning materials are used (Bonk & Graham, 2012;Güzer & Caner, 2014). It enables students to control the time, location, and paces of their learning to some extent (Güzer & Caner, 2014). The flipped classroom as a type of blended learning model used to promote student-centered learning and active learning (Pierce & Fox, 2012). It was originally used to provide videos or screencast instructions to students who were absent from class (Hamdan, McKnight, McKnight, & Arfstorm, 2013). Class time could be shifted from lecture-centered class to class with enriched activities to promote problem solving skills (Tucker, 2012). Flumerfelt and Green (2013) found using screencast videos in flipped classroom could promote the interaction between students and instructors, further fostering active learning (Leicht, Zappe, Messner, & Litzinger, 2012).

Personalized Instruction
Personalized instruction and learning have a theoretical base in learner-centered and constructivist learning (Reigeluth, Myers, & Lee, 2017;Watson & Watson, 2017). It customizes the instruction to individual learners' needs through providing learning resources, technologies, and activities (Kelly, 2016). Its learner-centered theory perspective can address learners' diverse backgrounds, competencies, and requirements (Green, Facer, Rudd, Dillon, & Humphreys, 2005). One way of personalizing the instruction is through technologies such as social bookmarking, blogs, and collaborative tools (Haworth, 2016). Besides technology-enabled personalized learning environment, social interaction and participatory learning also support personalization (Haworth, 2016;McLoughlin & Lee, 2010).

Research Design
A document analysis, a systematic approach to obtain meaning, understandings, and develop empirical knowledge by examining and analyzing the existing documents (Corbin & Strauss, 2008;Rapley, 2007), was the basis of the research design in this study. Such documents for review could include texts, images, and videos generated without a researcher's intervention (Bowen, 2009). Document analysis is a social research method and research tool (Bowen, 2009). This study adopted document analysis because the documents on the MIT website presented instructor's pedagogical strategies.

Data Collection
The documents reviewed were instructor insights of 15 MIT OCW Courses (table 1) from the Department of Electrical Engineering and Computer Science which was published on MITOCW site (https://ocw.mit.edu/courses/instructor-insights/#electrical-engineering-and-computer-science). In general, each instructor insights page included seven sections: (1) Course overview: a general information about the course; (2) course outcomes: course overall goals and learning objectives; (3) instructor insights: Instructors' thoughts on effective teaching strategies that they used; (4) curriculum information: semester of the course and other related courses; (5) assessment: the detailed assessment methods and percentage of each element; (6) student information: the number of students, their grade levels, and majors; and (7) how student time was spent: estimated time for students to spend on learning the course content in and out of class.
Nine courses were at the undergraduate level courses, and the remaining six courses were graduate level. Six of the courses provided text insights on the website, and eight provides video insights with verbatim transcripts. However, one graduate level course did not include detailed instructor insights. All the courses were taught with a team of educators (instructors, lab staffs, and teaching assistants). However, not all of the educators shared their insights. A majority of the insights are from one or some of the instructors from each course.

Data Analysis
The data were analyzed using thematic analysis (Braun & Clarke, 2006;Braun, Clarke, & Rance, 2014), following the procedures of Braun and Clarke's (2006). The steps are: (1) becoming familiar Open Praxis, vol. 12 issue 2, April-June 2020, pp. 257-270 with the data; (2) inductive open coding; (3) identification of themes; (4) review of themes; (5) refining and defining themes; and (6) report writing. Following a review of the 15 courses' instructor insights, the data for each course was coded. The analysis unit was the meaning unit. Themes were identified across all the courses. To increase the trustworthiness of this study, a debriefing was held with an expert in OER field.

Findings: Research Question 1 (RQ1): What are the Effective Pedagogical Strategies for MIT OCW EE and CS Courses?
A variety of pedagogical strategies were shared by the faculty members regarding teaching EE and CS courses. They included active learning, personalizing instruction, engaging learners, providing feedback, building learning community, clarifying learning objective, and integrating teaching and research.

Active Learning
Active learning is one of the most popular pedagogies used by the MIT OCW instructors. It means that learners should be actively involved in learning through hands on activities and interactions, rather than just passively listening to the lecture. Active learning is an umbrella which covers a variety of strategies such as hands on experience, authentic problems, flipped classrooms, discussions, think-pair-share, debates, and etc. One of the active learning approaches that were shared by MIT OCW instructors was practice-theory-practice, which means that instruction should expose students to practice first, followed by presenting theory and providing practical problems to solve. Similarly, Dr. Blade Kotelly, an instructor for Engineering Innovation and Design also emphasized hands on experience in his course. As he said: The lectures are interspersed with activities. So students will do some hands-on activities every, let's say, half hour. Probably at the limit is about halfway through, about an hour through, they'll have to do something no matter what. Because you want to keep students' attention up.
Virtual learning environment was used for hands on activities. Dr. Chris Terman, an instructor of Computation Structures shared these thoughts on providing virtual lab for hands-on activities: These virtual labs actually… takes courses from being a listening experience with maybe some pencil P-sets to your hands are active. So hands-on, brain on, right? And when people's brains turn on, it's amazing what they remember.
Another active learning strategies was flipping the classroom. Dr. Sanjoy Mahajan, a visiting professor of MIT, stated "we flip the classroom in the sense that we have students do as much as they can outside of class to climb the active learning hierarchy so that when we're together in class we can focus on constructive and interactive learning." A common active learning method used by MITOCW instructors is discussion, which is also an important element in flipped classroom. Dr. Katrina LaCurts, the instructor of the course, Computer System Engineering, said: Each recitation focuses around a technical paper that the students have read beforehand. And the goal for those discussion sections, those recitations, is for them to be largely discussion based… A large part of that instruction happens in the recitation, where we're having these discussions. And they go and apply those skills to their design projects.
The detailed discussion formats vary. One of the commonly used discussion formats as well as an active learning method is think-pair-share. Dr. Janet Rankin, the Interim Director of the Teaching and Learning Lab at MIT at the time of the interview, explained: In general, you first give students time to think about a question or a situation or some other scenario that you want them to think about. And you give them time to think actively but alone about that. And often you can ask them to write down their thoughts. But then at the end of that short period of time, whether it's three minutes or five minutes, you have them pair up. And if it's a really big class or the numbers of students in your class warrant it, you can have them triple up, it doesn't have to be a pair.

Personalized Instruction
Another important pedagogy was personalizing the instruction, which engages learners with diverse knowledge levels and background. It may be invoked via customized language use, optional learning materials, and offering learning materials online. Also, the use of different languages is productive in this regard. Dr. Dennis Freeman, based on experiences of addressing students with diverse prior knowledge, concluded: When I'm talking to a student in lab, I adjust what I'm saying to match the student's current level of understanding. If the student has never seen programming before, I don't use advanced programming comments in the way I talk to them. If they've never seen a circuit before, I don't use jargon. I've learned how to say things without using jargon.
Personalized instruction requires optional learning materials. For example, Dr. Chris Terman, provided a range of materials to benefit students with different backgrounds and learning preferences: So I create a huge--I think of it as a buffet. There's lots of dishes. And you can start at the beginning of the buffet and sort of pick it up from scratch. Or you can say, I'll skip the first couple courses, and I'm ready to dive in sort of in the middle of the conversation somewhere.
The short and small asynchronous materials available online offer students an opportunity to study at any time and any place. Dr. Chris Terman noted: It's keeping things short and sweet. So you have a huge--because now you have a bunch of short bites. Now, the MITx platform lets you organize those with questions that let you sort of continue to test your learning. So it's actually worked out to be a very nice way of making a fairly organized tour through the material that the students can start, and stop, and come back to. Plus, it's asynchronous.
In other words, they get to choose their time and place.

Engage Learners
Another theme was learner engagement. Relevant strategies include storytelling, joking, showing passion, and making class fun. Dr. Chris Terman emphasized the importance of engaging students, especially, after a long talk on technical topics. Dr. Patrick Winston, who taught Artificial Intelligence used stories to provide a big picture and inspire learners: I think stories are an important element of education, and if you strip them out, you don't have much left that can possibly be inspiring… I call them powerful ideas. If all you're teaching is skills, the educational experience you offer students is okay, but if you can accompany the skills with some big-picture, powerful ideas, the educational experience becomes more impactful, more important.
Another way to engage learners is to convey passions, as Dr. Winston sated, There are things people need to know, but you can't say they're very exciting. You have to pretend they're exciting. Somehow. Otherwise, your passion won't come across and your teaching won't be inspiring… One of my colleagues told me that he always ends his lectures with something fun so that people feel like they've enjoyed the class the whole time. It could be a joke, or an historical anecdote, or an intriguing demo. I do that now, too. I always try to end with something fun.
Guest speakers provided diverse perspectives and engaged learners. Dr. Joel Schindall, who taught Engineering Innovation and Design said: In some cases, we'll bring in someone from the Engineering faculty who is particularly gifted at communicating mechanical engineering design skills or electrical or chemical, because we want to give the students--there tend to be some discipline unique ways of thinking, and we want to give the students an idea of what the broad range is.

Provide Feedback
Providing systematic support and feedback to learners was important. Usually, they have an education team including instructors, lab staffs, and teaching assistants (TAs). Dr. Chris Terman indicated the advantages of having a hierarchy feedback system: But the students actually prefer the other thing, which is actually asking an TA is not very intimidating. The students, maybe they just took it last semester... And then you sort of work the chain up, work up the hierarchy to get an answer of people below. And that way you're only asking questions of the more intimidating people when you're pretty sure that no one else has the answer.
In addition, TAs provided optional tutorials. Students who wanted to join the tutorials can sign up the session. A third of the students of Dr. George Verghese's, Signals, Systems and Inference, attended the weekly tutorials noted: The teaching assistants go prepared with a small set of basic problems, simpler than those on homework, and illustrating points that have come up in lecture. However, the tutorials are also teaching assistant office hours, and students are encouraged to come with questions they may have.
Serval courses used discussion forum as a platform for questions and answers. The instructors and the TAs provided timely feedback to address students' questions. When a student asks a question in the forum, other students with similar question benefit from the response. The online discussion forum scales up the way of providing feedback and enhances the efficiency of providing support to students. Dr. Terman said: For the first time I'm able to make a thoughtful answer to a question and have 180 people look at the answer instead of one. And then the next person who has the same question, you say, well, I just spent 10 minutes. And with a large class, you can't spend 10 minutes for each of 300 people.

Learning community
Learning communities provide a physical or virtual learning environment to support building learning community. Dr. Verghese created formal learning community through providing a physical learning space and TA support: We reserve a classroom for the three or four evenings that precede the day homework is due, and guarantee that at least one of the staffs will be present there for 1.5-2 hours; usually we have the lecturer or a recitation instructor, as well as a teaching assistant. We find students working individually as well as collaboratively, and periodically interacting with the staff, either at the board or at their deskvery immersed and engaged in the homework problems, and in sorting out ideas and misconceptions related to these. The staff will typically respond to student questions with other (well chosen!) questions or hints that guide them along, rather than with answers-and that makes for a very fruitful dynamic.

Learning objectives
Some instructors proposed to share learning objectives with learners at the beginning of the class. Dr. Patrick Winston said: "you want to tell them what they'll be able to do at the end of the lecture that they couldn't do in the beginning. I try to start every lecture with a promise, every time." Dr. Philip B. Tan, who taught Creating Video Games, shared similar ideas: So one thought that I have for educators who are running those classes is be very clear to yourself and then to the students about whether you are running a class about game design, or game programming, or project management. When we started this class, we were trying to be as clear as possible to the students. This is a class project management. You will do all those other things in the process of this class, and many of you are here for that reason.

Integration of Teaching and Research
Integrating research into teaching stimulates instructor motivation and engages learners. Some instructors used the research results in their teaching, and were inspired to pursue related research topics. For instance, Dr. George Verghese, who taught Signals, Systems and Inference, noted regarding integrating teaching and research: And all of my research is also around algorithms. So this is me living the dream, teaching the topic that I love. And it's an exciting class… I try to add in new topics that I don't know so well, so I learn them even better. And that, in turn, influences my research.

Findings: RQ 2: What are the Commonly Used Assessment Methods of MIT OCW EE and CS Courses?
Formative assessment included quizzes and oral exams for just-in-time teaching, as well as the creation of an online automatic environment for students' self-assessment. The summative assessments were primarily final exams and projects.

Formative Assessment
Formative assessments were widely used by MIT OCW instructors. Some instructors used in class formative assessment to get to know students' learning levels to adjust their teaching accordingly. For instance, Dr. Blade Kotelly stated, "we administer a quiz, we swap all the quizzes, we review all the answers… So we try to do a diagnostic to see what's happening." Similarly, Dr. Dennis Freeman used formative assessment for just-in-time teaching: Since implementing the practice-theory-practice approach, I've become more careful to assess students' understanding during lectures. I do this by asking a concept question every 15 minutes or so... Students work in pairs to answer the questions. I show them five possible answers, and they raise their hands showing some number of fingers that corresponds with their answer choice. I look at their responses. If everybody gets the question right, I know I don't need to explain the concept again. I keep going. If some students get the question wrong, I provide more explanation.
If everyone gets the question wrong, then I know I didn't explain the concept well and I start from the beginning.
They also used technology to enhance the efficiency of providing feedback. Dr. Sanjoy Mahajan described noted students in the design lab put their questions in an electronic help queue seen by all the students and TAs. TAs could use their mobile phone to respond, provide check-off points, and record the results online. One of the methods to check whether student master the learning content was to use oral exams. However, its effectiveness was limited due to the large number of evaluators. Dr. Freeman noted: We asked very open-ended questions, from which we learned a lot. We would start with an easy question, a question that we expected everybody would get. If the student got it easily, then we would ask a harder question. If the student didn't get it easily, then we'd ask another easier question. If he or she sailed through the easy question, we went straight to a difficult question. In other words, we adjusted to the student's level, and it was really quick. In 10 minutes, and sometimes fewer, we had a good feeling for the student's level of understanding.
Self-assessment was obtained via an online tutorial environment in which students checked their code, and receive feedback. If incorrect, detailed feedback is provided. Dr. Dennis Freeman said: "Checking test cases offers much richer feedback. 'Your code passes tests 1, 2, 4, and 5 but fails test 3.' This message contains a wealth of information about not only the problem at hand but also about how to construct effective test cases, which is essential to becoming an expert programmer."

Summative Assessment
Final exams, projects, and presentations were the basis of summative assessments. Projects were used in Dr. Erik Demaine's graduate level course Algorithmic Lower Bounds: Fun with Hardness Proofs: So with every sort of advanced class that I teach there's a final project. And the goal of the final project is for students to somehow get their feet wet with the material and sort of experience it at a more researchy level. In general, this can be things like surveying papers that I didn't cover in the class because there's only so much you can fit in one semester. So they'll go and read other material and kind of aim to teach that to the students. So there's a written project part, and then there's also a presentation in class. So this is an opportunity for students to learn more.

Findings: RQ 3: What Challenges do the Instructors Perceive They have While Teaching MIT OCW EE and CS Courses?
Challenges ranged from ways to assess students' learning to approaching in changing other instructors' pedagogical beliefs. Dr. Erik Demaine explained the challenge to assess whether students could apply the algorithms creatively: But it's hard to measure a student's understanding because it's like, did you get the creative trick that we had in mind, or find another one that's just as good? Students may find a different creative trick that doesn't end up with as good an algorithm in the end. So we penalize that someone, but for the most part we are happy when people get correct algorithms.
Regarding colleagues' pedagogical beliefs, Dr. Freeman stated: In fact, it's been a process to shift the teaching model toward a hands-on approach, such that most of the learning happens in the lab. It's been hard to get the faculty on board with this shift. Some of our most effective lecturers have commented, "I don't want to teach this course because there's no teaching," which is completely wrong. Usually by the time they've done it a few times, faculty realize that they're actually imparting a lot more knowledge by facilitating hands-on learning than they would solely by lecturing. There's a misconception on the part of "broadcast" lecturers that if they say it, students will understand it. That's so wrong.

Discussion
The purpose of this study was to examine MIT OCW STEM instructor insights, particularly their pedagogical strategies, assessment methods, and perceived challenges in instruction in order to inform other STEM educators regarding effective pedagogical strategies and assessment methods. Fifteen MIT OCW instructor insights from the department of Electrical Engineering and Computer Science were reviewed. A variety of pedagogical strategies were identified, such as active learning, personalizing instruction, engaging learners, providing feedback, building learning community, clarifying learning objective, and integrating teaching and research. Both formative and informative assessment methods were used. Challenges such as effective way to assess learners and changing instructors' pedagogical beliefs were identified.
Active learning is one of the primary themes of the instructor insights. Flipped classroom is one of the effective active learning strategies used by instructors, which concurs with previous research (i.e., Freeman et al., 2014;Michael, 2006;Prince, 2004;Lund & Stains, 2015). The flipped classroom manifests the core pedagogical shift from lecture-centered teaching to learner-centered instructions that use activities to engage learners and solve problems (Tucker, 2012). However, the pedagogy shifts also encountered barriers as identified in this study. Instructors doubted the effectiveness of active learning strategies and felt uncomfortable of using it. This finding is in line with the previous research which indicated that barriers hinder the adoption of active learning strategies in classroom (Finelli et al., 2014;Froyd et al., 2013;Lund & Stains, 2015;Shadle et al., 2017) and instructors do not believe in the effectiveness of active learning (Tharayil et al., 2018). Thus, one of the possible ways to address this problem is to provide professional development to educate instructors by showing successful examples of using active learning.
Another important pedagogical strategy is personalized instruction reported by MIT OCW instructors. This study found that the common practice of personalization is through providing optional materials to learners. These personalized instructions address needs of individual learners (Kelly, 2016) with diverse backgrounds, competencies and requirements (Green et al., 2005). As more and more blended learning mode is used in higher education, providing optional learning materials online could be one way to provide personalized instruction in higher education.

Limitations and Future Research
There are several limitations in this study. First, the researcher only reviewed MIT OCW instructor insights. If the researcher could also review the learning materials such as videos and quizzes of the course to triangulate the data, it would possibly increase the trustworthiness of the study. Future researchers could combine both instructor insights and MIT OCW course learning materials for data triangulation. In addition, the pedagogical strategies were from educators' perspective, future research might interview learners who have taken MIT OCW STEM courses regarding their learning perceptions and experiences of effective instructional strategies.

Conclusion
The significance of this study contributes to both research and practice. For research, this study indicated that a variety of pedagogical strategies could be used for STEM education such as active learning, personalizing instruction, and etc. In addition, formative assessments such as quizzes and in class evaluations could be used for just-in-time teaching to improve the teaching quality; and both final exams and projects could be used for summative assessment in STEM education. This study indicated that the majority of the pedagogical strategies and assessment methods were aligned with the pedagogy of student-centered learning. This study could be an initial step of the research on the impact of OER on STEM educators teaching pedagogies.
For practice, STEM instructors and instructional designers could leverage the existing experience of MIT OCW STEM educators to improve teaching and student learning. Given challenges faced on how to effectively and efficiently evaluate learning and changing faculty members' teaching beliefs, practitioners could keep these challenges in mind while designing and delivering their instructions and figure out strategies to address these issues.