Did Experience with Digital Instructional Materials Help Teachers Implement Remote Learning During the T

C O R P O R A T I O N T he coronavirus disease 2019 (COVID-19) pandemic forced educators to rapidly adapt to remote learning during spring 2020, resulting in a scramble for teachers to ensure that students had remote access to instructional materials. Previously, principals reported that the ability to deploy online curricula was not a limitation. However, they indicated a relatively high need for high-quality materials to support academic instruction while school buildings were closed. Nearly half of teachers reported a need for “strategies to keep students engaged and motivated to learn remotely” (Hamilton, Kaufman, and Diliberti, 2020, p. 9). Prior work also established that digital instructional materials were used widely before the pandemic, although teachers typically used them to supplement comprehensive curriculum materials (Tosh et al., 2020). Although 88 percent of U.S. teachers used digital materials during classroom instruction in the 2018–2019 school year, fewer than 20 percent of teachers reported using any one of their digital materials for more than half of their instructional time. Furthermore, of all of the materials that teachers named as their main materials before the pandemic, only 30 percent were digital materials (Tosh et al., 2020). Understanding how Key Findings

digital materials were used leading up to school closures can shed light on the extent to which teachers were prepared to pivot to full virtual learning. This also might identify areas in which additional supports could be helpful in using digital materials for both virtual and in-person instruction.
In this Data Note, we examine teachers' use of digital and comprehensive curriculum materials during the 2019-2020 school year prior to COVID-19 disruptions and teachers' perceptions about the extent to which their digital materials connected to their main curricula. Finally, we consider how teachers' use of digital materials prior to the pandemic was connected to teaching and learning during school closures in spring 2020. We draw on data from the spring 2020 American Instructional Resources Survey (AIRS), which was fielded in May and June 2020 to a nationally representative sample of teachers and school leaders who are part of the RAND Corporation American Educator Panels (AEP) and to state-representative samples of teachers in 12 states. This Data Note presents a small, focused set of key findings from the teacher responses; it omits some potentially valuable findings from the full set of survey questions. 1

Majorities of Teachers Reported That the Main Materials That They Used Prior to the Pandemic Provided Digital Instructional Materials
To understand how prevalent digital instruction was in the classroom before the pandemic, we asked teachers whether their main materials provided digital instructional materials for all students. In selecting their main materials, teachers could include comprehensive curriculum materials or additional instructional 1 A full set of survey results and technical documentation, including school leader responses to similar questions about English learners, is provided in Doan et al., 2020; the data are available to download from the AEP data portal to enable others to conduct analyses (RAND Corporation, undated).

Definitions of Key Terms
• Digital materials, for the purpose of this study, are instructional materials that are available online for teachers and students that do not constitute a full course of study. These exclude comprehensive curriculum materials that are available in online form (e.g., EngageNY). Some commonly used digital materials are Kahoot! and Quizlet, which are websites where teachers can generate quiz-like games; ReadWorks and NewsELA, which provide online access to articles and question sets by grade or reading level; and BrainPOP and Khan Academy, which present videos and instructional resources for those videos.
• Comprehensive curriculum materials are instructional materials that are intended to constitute a full, comprehensive course of study for a particular subject and grade level. Comprehensive curriculum materials are available in print and/or online form.
• Main materials are materials that teachers indicated using regularly as the two to three materials that they use most. These could include comprehensive curriculum materials or additional materials (digital or nondigital) that do not constitute a full course of study.
• Supplemental materials are materials that teachers reported using for less than half of their instructional time and/or did not report using as main materials.
materials-both digital and nondigital-that do not constitute a full course of study. Seventy-seven percent of teachers somewhat or strongly agreed that their main prepandemic materials provided digital instructional materials (see Figure 1). We also explored whether particular subgroups of teachers were more likely to indicate that their prepandemic materials provided digital instructional materials. In every grade band, more than three-quarters of teachers agreed that their main prepandemic materials provided for digital instruction with slightly higher, although significant, percentages in middle school (82 percent). We found similarly modest differences by school urbanicity, where suburban teachers (79 percent) were significantly more likely to say that their main materials before the pandemic provided digital instruction than were their counterparts teaching in cities (74 percent). 2 We did not find differences by school poverty level, school racial or ethnic composition, or teachers' use of standards-aligned materials for whether teachers' main prepandemic materials provided digital instruction.

Most Teachers Used Digital Materials for Less Instructional Time Than Their Core Curricula
We separated teachers' instructional materials into different categories-digital materials that are available online that do not constitute a full course of study and comprehensive curricula-and asked about their use during the portion of the 2019-2020 academic year prior to the COVID-19 pandemic. We found that 90 percent of teachers reported using digital instructional materials once per week or more during classroom instruction, with 44 percent considering a digital material to be one of their main materials, which is consistent with prior findings. However, most teachers reported that digital materials occupied a minority of classroom time: Only 38 percent of teachers reported using a digital material for half or more of their instructional time during the 2019-2020 academic year. This indicates that teachers might have used digital

Percentage of Teachers Reporting Whether Their Main Instructional Materials Provide Digital Instructional Materials
NOTE: This figure shows levels of teacher agreement with the statement, "My main instructional materials provide digital instructional materials for use by all students." Percentages presented are inclusive of teacher responses across grade levels (elementary, middle, and high) and subject area (English language arts [ELA], mathematics, and science).

Strongly disagree
Somewhat disagree Somewhat agree Strongly agree 7 16 33 44 main curricula and observed several patterns across subjects. First, online resources supporting the use of student learning standards, such as the Common Core State Standards Initiative website and the Next Generation Science Standards website, and teachers' state department of education websites were most frequently noted by teachers as fully connecting to their main curricula. Of the teachers who stated that they used these materials, roughly two-thirds of ELA and math teachers responded that the Common Core State Standards Initiative and state department of education websites fully connected with their main curricula, and almost three-quarters of science teachers stated that the Next Generation Science Standards website fully connected with their main curricula ( Figure 2). Notably, these are digital materials that teachers referenced to plan their instruction rather than materials that teachers used for instruction itself.
Among the digital materials used in classroom instruction, teachers often noted that materials that are designed to provide content-specific, standards-aligned instruction, such as Lexia, i-Ready, ixl.com, Khan Academy, ST Math, and PHeT Interactive Solutions, were connected with their curriculum. About half to almost two-thirds of the teachers who used these materials stated that they fully connected with their main curricula; however, these digital instructional materials were not widely used by teachers ( Figure 2). In comparison, some of the most commonly used digital materials-Teachers Pay Teachers, Google, YouTube, and Kahoot!-were largely cited by teachers as not connecting fully with their main curricula. Only about one-third of the ELA teachers who used these materials perceived them as connecting fully with their main curricula, and we saw similar results for math and science teachers ( Figure 3).
Coherence among teachers' instructional materials is especially important because it might have further supported teachers in holding students accountable during remote learning. Teachers who reported using at least one digital material that fully connected with their main materials were significantly more likely to send schoolwork home to materials in a more supplementary role. However, we saw a substantial increase in the percentage of teachers reporting that they use a digital material for half or more of instructional time-20 percent in the 2018-2019 school year compared with 38 percent in the 2019-2020 school year (Tosh et al., 2020). This increase could be attributed to a broader trend in which teachers are devoting more instructional time to digital materials. Alternatively, it is possible that some teachers responded about their experience using digital materials during school closures even though our survey asked about digital materials prior to the pandemic. Nevertheless, although increasing shares of teachers incorporate digital materials into their classroom instruction, a majority of teachers were still using digital materials in a supplemental fashion (according to the defini tion of supplemental materials we present in this Data Note).
The AIRS distinguishes between materials used to plan instruction and materials used as part of classroom instruction. Across subjects, the most commonly used digital materials to plan instruction were Teachers Pay Teachers (54 percent) and search engines (52 percent). Among the digital materials that were used most commonly during instruction, teachers reported a mix of cross-subject materials (e.g., YouTube and Kahoot!) and materials designed to provide content-specific or standards-aligned instruction (e.g., Khan Academy and Newsela), 3 which is consistent with what teachers reported using in spring 2019 (Tosh et al., 2020).

The Digital Materials Most Commonly Used Before the Pandemic Did Not Fully Connect with the Content, Concepts, or Activities of Main Curricula
We asked teachers about the extent to which their prepandemic digital materials connected clearly with the content, concepts, and learning activities of their  students during remote learning with the expectation that students would complete the work (74 percent) compared with teachers who did not use at least one digital material that fully connected with their main materials (65 percent). These teachers also were significantly more likely to grade the work that students completed during remote learning (57 percent versus 51 percent) and were significantly more likely to provide feedback to students daily or more often during remote learning (40 percent versus 35 percent). COVID-19 pandemic in spring 2020. We hypothesized that teachers who agreed that their main prepandemic materials provided digital instructional materials might have been better positioned to pivot to remote learning because of (1) greater familiarity with digital instruction on the part of teachers and students and (2) the potential ability to use their prepandemic main materials during remote learning because those materials already provided digital instructional materials. To understand this relationship, we looked at whether instructional practices pattern was consistent across subject areas. However, we did not find a relationship between the extent to which teachers reported that their main prepandemic materials provided digital instructional materials and how often they provided grades and feedback to students during remote learning. 5 We also did not find a relationship between whether teachers' main prepandemic materials provided digital instructional materials and their expectations about whether schoolwork would be completed during remote learning. Furthermore, we did not find a relationship between whether teachers' materials provided digital instructional materials and the percentage of students completing assignments. related to coursework, hours of planning or instruction, and communication with students and families during school closures varied based on teachers' reports of whether their main prepandemic materials provided digital instructional materials. 4 One dimension of remote learning we examined was the extent to which teachers provided grades and feedback to students. Teachers who strongly agreed that their main prepandemic materials provided digital instructional materials for use by all students were significantly more likely to grade the work completed during the time school was closed because of the pandemic (60 percent) compared with those who strongly disagreed (43 percent) (Figure 4). This NOTE: This figure shows the percentage of teachers agreeing that "My main instructional materials provide digital instructional materials for use by all students" and whether they graded work completed by students during remote learning in spring 2020. The responses do not sum to 100 percent within each response category because teachers were also able to choose the response option "varies based on subject or class." The following comparisons were significant at the p < 0.05 level: strongly disagree versus somewhat agree, somewhat agree versus somewhat disagree, strongly agree versus somewhat agree, and strongly agree versus somewhat agree. Next, we examined whether there was any relationship between whether teachers agreed or disagreed that their main prepandemic materials provided digital instructional materials and the number of hours during remote learning that teachers spent planning or the number of hours that students were expected to engage in learning activities. We hypothesized that the use of instructional materials that provided digital instructional materials, compared with the use of materials that did not, might have eased the transition to remote learning and reduced the number of hours that teachers spent planning. Additionally, we hypothesized that the use of instructional materials that provided digital instructional materials might have increased the number of hours of learning activities that students were asked to complete during remote learning because both teachers and students would have had more-ready access to or experience with online learning, allowing for a greater workload. However, the number of hours that teachers spent on instructional planning once school closures began did not differ according to whether their prepandemic materials provided digital instructional materials, and neither did the number of hours of learning activities that teachers asked students to complete on a typical weekday during school closures. These findings suggest, first, that the transition to remote learning was so monumental a shift that even prior familiarity with digital materials did not reduce the number of hours that teachers had to spend on instructional planning. Second, these findings suggest that, although teachers whose main prepandemic materials provided digital instructional materials were more likely to provide grades during remote learning, they did not require different quantities of schoolwork from students.

Discussion and Recommendations
Teachers whose main prepandemic materials provided digital instructional materials might have leveraged their experience to hold students accountable during remote learning, as they were more likely to grade students' work. It is possible that these

Math Teachers Whose Main Prepandemic Materials Provided Digital Instructional Materials Were More Likely to Engage in Synchronous Communication During Remote Learning
We examined whether teachers' prior use of digital instructional materials was related to how they communicated with students during remote learning. We hypothesized that teachers with greater prepandemic familiarity with digital instructional materials would engage in more synchronous communication than their counterparts because of readily accessible online curricula and tools and greater comfort using those digital resources. In the survey, we separately asked teachers about the frequency of their synchronous and asynchronous communication with students. 6 We found that math teachers who strongly agreed that their main prepandemic materials provided digital instructional materials for use by all students were significantly more likely to engage in frequent (daily or more often) synchronous communication with their students (27 percent) compared with 16 percent of teachers who strongly disagreed with the statement. 7 However, we did not observe this pattern among science and ELA teachers.
On the other end of the spectrum, we asked about teachers' use of asynchronous communication during remote learning. ELA, math, and science teachers who strongly disagreed that their main prepandemic materials provided digital instruction were significantly more likely to state that they engaged in asynchronous communication with students daily or more often (66 percent) compared with teachers who somewhat disagreed (51 percent) and somewhat agreed (53 percent). faced myriad challenges during remote learning, including students' lack of access to devices and the internet, teachers' difficulties in contacting students, and families' economic and health concerns, all of which influenced these outcomes and could not be addressed by teachers' prior experience with digital materials alone. Teachers and students need resources and systems that address the full challenge of remote learning, beyond access to and support for using digital instructional materials.
Given indications from other RAND research that remote learning-in the form of virtual schools, online options, or hybrid forms of instruction-could continue even after the pandemic is over (Schwartz et al., 2020), these findings provide several opportunities to improve remote learning. The pandemic emphasized the need for flexibility in teachers' use of instructional materials and coherence among the many different types of materials teachers use. To achieve this and provide for more-robust instruction in the remote learning setting, curriculum developers should consider how to embed digital instruction into curricula. Developers of supplemental digital instructional materials should pay particular attention to how their products align with the content and concepts of commonly used curricula. Policymakers and districts can expand access to and promote selection of such materials and support teachers' and students' use of those materials so that students and teachers in remote learning environments are poised for greater success.

Limitations
This Data Note presents findings on digital instructional materials that can be used to improve remote learning and in-person digital instruction. However, it is important to note a few limitations.
First, our descriptive analyses relied on teachers' self-reports of whether their main instructional materials provided access to digital instructional materials for all students. Therefore, these data do not reflect whether teachers' main materials provided digital instructional materials as assessed by curriculum developers, curriculum reviewers, or research. teachers continued using the same materials that they used prior to the onset of the pandemic because these materials already contained digital elements, which maintained consistency for students. These teachers-and their students-also might have had more familiarity with and knowledge of how to use digital materials, thereby increasing teachers' confidence in assigning grades and providing remote instruction. Additionally, these teachers were less likely than those whose main prepandemic materials did not provide digital instructional materials to rely on frequent asynchronous communication with their students, which might be less engaging than synchronous communication. However, our data do not tell us whether the students of these teachers were any more successful, academically or otherwise, than their peers during remote learning.
To provide students with a coherent learning experience, the digital materials that teachers use ideally should connect well with teachers' main curricula. However, some of the most commonly used digital materials were noted by teachers as not connecting fully with their main curricula. Instead, the digital materials that teachers cited as connecting fully with their main curricula included those that were used only for planning purposes or the less commonly used digital instructional materials that are designed to provide content-specific and standards-aligned instruction. Consequently, during remote learning, teachers might have been grading students on materials that were not aligned with standards and were unconnected to their curricula. On the other hand, teachers who did report using at least one digital material that fully connected to their main prepandemic materials were more likely to hold students accountable during remote learning.
Prior experience with digital instructional materials did not ease teachers' transitions to remote learning during school closures in all of the ways that one might have expected. Teachers whose main prepandemic materials provided digital instructional materials were not able to spend less time on instructional planning, and their students were not more likely to complete assignments on a regular basis. Teachers, students, and families unobserved school or classroom differences or for other contextual factors, such as district or state remote-learning policies. Therefore, our findings should be interpreted as associational and not causal.
Third, the AIRS was fielded during school closures because of the COVID-19 pandemic. Experiences with remote instruction might have influenced teachers' responses and, therefore, our findings. In particular, teachers might have responded about their experience using digital materials during school closures even though our survey asked about digital materials prior to the pandemic.
Teachers' reports might be biased by their overall perceptions of their materials, and teachers might have varying definitions of what constitutes digital instruction. Although teachers' self-reports offer an important window into the prevalence of digital instructional material use in the classroom, educators and policymakers should consult independent assessments of the "digitalness" of instructional materials when making curriculum decisions.
Second, although we tested the robustness of our descriptive patterns by controlling for teacher and school characteristics, we cannot account for

How This Analysis Was Conducted
In this Data Note, we use responses from 5,978 teachers to the 2020 AIRS to examine teachers' use of digital instructional materials. 1 Specifically, we focused our analysis on the following categories of items from the 2020 AIRS: Items relating to teachers' use of instructional materials prior to school closures. These items were prefaced with the following guidance: "If your school has closed because of COVID-19 concerns, please respond in regard to curricula and other instructional materials used before your school closed." The items are as follows: • Beyond curricula, please indicate which additional instructional materials you use regularly (once a week or more, on average) for instruction this school year (2019-2020). • Beyond curricula, please indicate which additional materials you reference regularly (once a week or more, on average) to plan your instruction this school year (2019-2020). • Of all the instructional materials you use, please indicate approximately what percent of instructional time you dedicate towards using them for a typical class of students each week. • Of the instructional materials you use, please choose the one main material you use the most. If there is not one main material you use most, or you use different main materials for different sets of students you teach, choose 2-3 main materials you use most.
Items relating to teachers' perceptions of instructional materials, including the extent to which they connect to main curricula and the extent to which they provide digital instructional materials. These items were prefaced with the following guidance: "If your school has closed because of COVID-19 concerns, please respond in regard to curricula and other instructional materials used before your school closed." The items are as follows: 1 Participants chose from a list of commonly used digital materials, excluding digital materials that included comprehensive curricula. We compiled the list of digital materials included in the survey based on teachers' responses in previous American Teacher Panel surveys to similar questions about which digital materials they use. Teachers were asked about their use of digital materials for only one subject area (ELA, mathematics, or science), based on their response to a question at the beginning of the survey about which subject(s) they taught. If they reported teaching more than one subject (ELA, mathematics, or science), they were randomly selected to complete the survey for only one of those subject areas.
• For the instructional materials you use beyond curricula, how would you rate the extent to which each connects clearly with the content, concepts, and learning activities in the main curricula you use? • Indicate your agreement or disagreement with the following statements about your main materials, excluding the materials that you have created yourself: My main materials provide digital instructional materials for use by all students.

Items relating to teachers' instruction during school closures:
• Have you graded the work being completed during the time school has been closed?
• In a typical week since your school has been closed, approximately how many hours have you spent on instructional planning, including creating lessons, materials, or ensuring online access for students? • While your school has been closed, how frequently have you engaged in the following types of communication with your students? Teachers responded separately for asynchronous communication, such as email, text, and/or postings to a learning management system; synchronous communication, such as phone or video calls (e.g., Zoom, Google Meet); and feedback or grades for students on their work. • Please estimate the proportion of your students who are completing assignments you have provided during the time your school has been closed. • Approximately how many hours of learning activities have you asked your students to complete on a typical weekday during the time your school has been closed?
Throughout this Date Note, we report sample-wide and subgroup-specific means and proportions of variables of interest, weighted using a set of nationally representative weights described in further detail in the American Instructional Resources Survey Technical Documentation (RR-A134-4, www.rand.org/t/RRA134-4). To compare responses for teachers in schools with different demographic profiles, we matched AIRS responses to school-level data from the 2018-2019 Common Core of Data and examined differences across school enrollment of students eligible for free and reduced-price lunch, school enrollment of students of color, and school urbanicity (city, suburban, town, rural).
We conducted a series of supplemental regression analyses to assess whether differences in teachers' perceptions about whether their main materials provide digital instruction for all students persisted when including statistical controls for select teacher and school characteristics. Results across these specifications were substantively similar to trends present within descriptive subgroup comparisons; therefore, we present only the simple weighted means and proportions in this Data Note.
After conducting our regression analyses and noting significant differences, we performed adjustments to correct for multiple hypothesis tests using the Benjamini-Hochberg correction for multiple comparisons.

Key Recommendations in This Report
To provide students with a coherent learning experience, the digital materials that teachers use ideally should connect well with teachers' main curricula.
Teachers and students need resources and systems that address the full challenge of remote learning, beyond access to and support using digital instructional materials.
Curriculum developers should consider how to embed digital instruction into curricula.
Developers of supplemental digital instructional materials should pay particular attention to how their products align with the content and concepts of commonly used curricula.
Policymakers and districts can expand access to and promote selection of such materials and support teachers' and students' use of those materials so that students and teachers in remote learning environments are poised for greater success.

About This Report
The American Educator Panels (AEP) are nationally representative samples of teachers and school leaders across the country. We are extremely grateful to the U.S. public school teachers and leaders who have agreed to participate in the panels. Their time and willingness to share their experiences are invaluable for this effort and for helping us understand more about how to better support their hard work in schools.

RAND Education and Labor
This study was undertaken by RAND Education and Labor, a division of the RAND Corporation that conducts research on early childhood through postsecondary education programs, workforce development, and programs and policies affecting workers, entrepreneurship, financial literacy, and decisionmaking. This report is based on research funded by the Bill & Melinda Gates Foundation, the Charles and Lynn Schusterman Family Foundation, and the Overdeck Family Foundation. We are grateful to the foundation staff for their collaboration and feedback on our surveys and analysis. The findings and conclusions we present are those of the authors and do not necessarily reflect positions or policies of the foundations funding this report.
More information about RAND can be found at www.rand.org. Questions about this report should be directed to ktosh@rand.org, and questions about RAND Education and Labor should be directed to educationandlabor@rand.org.

About the AEP Data Note Series
The AEP Data Note series is intended to provide brief analyses of teacher and school leader survey results of immediate interest to policymakers, practitioners, and researchers. If you would like to know more about the dataset, please see the American Instructional Resources Survey Technical Documentation (RR-A134-4, www.rand.org/t/RRA134-4) for more information on survey recruitment, administration, and sample weighting. If you are interested in using AEP data for your own surveys or analysis or reading other AEP-related publications, please email aep@rand.org or visit www.rand.org/aep.
The RAND Corporation is a research organization that develops solutions to public policy challenges to help make communities throughout the world safer and more secure, healthier and more prosperous. RAND is nonprofit, nonpartisan, and committed to the public interest.
RAND's publications do not necessarily reflect the opinions of its research clients and sponsors. R ® is a registered trademark.

Print and Electronic Distribution Rights
This work is licensed under a Creative Commons Attribution 4.0 International License. All users of the publication are permitted to copy and redistribute the material in any medium or format and transform and build upon the material, including for any purpose (including commercial) without further permission or fees being required.
For more information on this publication, visit www.rand.org/t/RRA134-8.