Reviews of research funded by U.S. Institute of Educational Sciences: A case of reading development and instruction

Abstract Empirical research is a critical in the field of education, however, research funding sources are often hard to identify. In this study, we examined how National Center for Education Research (NCER), a branch of the federal Institute of Education Science (IES), has supported research on various areas of reading development and instruction. Our goal was to identify the major trends and issues in NCER reading research projects over the past 13 years by examining the general characteristics, participants, and research foci of the studies. To do so, we reviewed and coded 158 research projects based on existing content analyses about reading research. Findings indicate that the NCER has allocated approximately $397 million to 158 reading research projects from 2004 to 2016. Research projects under the Reading and Writing program were the most frequently funded, particularly studies focusing on comprehension and including participants in grades K-3. Recommendations are made for reading researchers seeking funding based on the major trends and issues identified in NCER studies.


PUBLIC INTEREST STATEMENT
The goal of this study was to examine how federal funding has been distributed in the field of reading research over the past 13 years. We reviewed the research projects funded by the National Center for Education Research (NCER), a branch of the Institute of Education Science (IES). Proposal abstracts of funded projects were retrieved through the IES website and coded to explore the major trends and issues. Coding schemes were established and refined based on existing studies about literacy instruction. From 2004 to 2016, NCER funded 158 reading research projects with approximately $397 million. Reading research projects have been funded most by the Reading and Writing program. The most frequent research focus was reading comprehension. Grades K-3 was frequently studied. Recommendations will help reading researchers to seek research funding as well as to stay current in the field of reading research.

Introduction
Empirical research has long been recognized as critical to the field of education; it provides teachers with new instructional strategies, helps with the development of fair and unbiased assessments, assists with the creation of new interventions for struggling learners, and much more. Literacy education research is particularly essential, as current data show that 31% of fourth-grade and 24% of eighth-grade children in the United States are performing below level on standardized reading assessments (National Center for Educational Statistics, 2016).
To develop and provide effective instructional strategies, it is important for literacy scholars to be aware of the current trends and foci of literacy research, which has been hard to determine due to the constant shift in trends and the large scope and volume of newly published articles. While content analysis papers report overview of the different foci, methodology, grade level, and theoretical perspectives of literacy research published in leading journals, due to prolonged publication lags between manuscript submission and its final publication, these review papers may not be able to capture the most current research trends in the field.
An alternate way of examining current research trends is to track how research funding is allocated. In our study, we are particularly concerned with the money provided by the US Government for educational research, as these federal grants often result in larger scale investigations and culminate with more publications in higher impact journals than those funded by private monies. Milesi, Brown, Hawkley, Dropkin, and Schneider (2014) examined three different biometric analyses about an NSF-funded STEM research program and found that the principal investigators (PI) of funded research outperformed other STEM researchers in the amount of publications produced and the ratings of said publications after funding was awarded. It was also determined that their papers were published in higher impact journals than those of their peers. These findings indicate that literacy researchers should strive to obtain federal monies, however, where can these be found?
To answer this question, we determined that the Institute of Educational Sciences (IES), the primary research arm of the U.S. Department of Education, has historically produced the most impactful reading research of all the funding agencies. We found evidence of this in our examination of the four issues of Reading Research Quarterly, a leading peer-reviewed journal, published in 2015; approximately 53% of the research articles published had received either public or private funding, and out of those, 78% were supported by IES. Clearly, IES provides a tremendous amount of funding for literacy research, and so in the current study we will focus our analysis on the literacy projects supported by this agency.
The IES is broken into four divisions, which include the National Center for Educational Research (NCER), the National Center for Special Education Research (NCSER), the National Center for Education Statistics (NCES), and the National Center for Education Evaluation and Regional Assistance (NCEE). We chose to only analyze studies funded by NCER because of the abundance of programs this division has for research related to typically developing learners, the population we are interested in for this review. According to Findings from the National Center for Educational Research: 2002Research: -2011 grants were awarded to 591 projects between 2002 and 2011, 66 of which were for education-based research training. NCER monies have been provided for research on the identification of effective instructional practices, programs, and interventions for improving reading ("Findings", 2011). This wide variety of research topics and large amount of available funding indicates that an examination of NCER grants would provide reading researchers with a comprehensive overview of the field of literacy research.
Based on this information, we decided to focus the present study on how NCER grants have supported studies from various areas of reading, and to identify the major trends and issues in reading over the past 13 years. To achieve this goal, our research was guided by the following three questions: (1) What are the general characteristics of the NCER reading research grants from 2004 to 2016?
(2) What trends are reflected in the characteristics of participants in NCER funded research projects from 2004 to 2016?
(3) What trends are reflected in the foci of reading research projects funded by NCER from 2004 to 2016?

Selection of proposals
To identify the research projects funded by NCER, we entered the keywords "reading" and "NCER" into the IES website search tool. We selected "Grant" as the funding type in order to ensure that the system would only retrieve proposal abstracts of projects that had received grant monies. The initial search yielded grant proposal abstracts submitted as early as 2002, the year IES was founded. However, projects before 2004 were excluded from this review because their abstracts lacked the detailed information needed for coding. Once these were eliminated, we ended up with 364 proposal abstracts that contained our keywords of "reading" and "NCER" from 2004 to 2016.
The initial set of 364 proposal abstracts were screened via the following criteria: (a) Proposals that focused exclusively on children with learning disabilities (e.g. deaf, autism, or ADHD) were excluded; (b) Proposals that did not include school-aged children were excluded; (c) Proposals that focused only on writing were excluded; (d) Proposals that did not contain sufficient information were excluded.
While we are aware that funded projects focusing on children with learning disabilities address important issues and have critical implications for educational practices, in the present review we chose to exclusively concentrate on studies related to the reading development of typically developing school-aged learners. It should be noted that research projects with an emphasis on struggling readers without identified learning disabilities were not eliminated, as reading difficulties are commonplace among beginning readers and not necessarily indicative of a learning disability. Ultimately, the above criteria and initial screening allowed us to eliminate 206 abstracts, leaving 158 abstracts to be coded for further analysis.

Coding of proposals
The coding scheme is comprised of two parts, the first of which focuses on basic information retrieved directly from the IES website, including the title of the proposal, program name, goal, funding year, funding period, funding amount, grantee, and principal investigator (PI). The second part includes the pertinent information from the proposal abstracts, such as: (a) participant age (early childhood, middle childhood, early adolescent, and late adolescent), (b) special populations (English language learners and low SES), and (c) research foci (beginning reading, word recognition, vocabulary, fluency, reading comprehension, content area literacy, affect, technology, teacher education, and measurement).
The second part of the coding scheme followed the structure of previous content analyses on reading and literacy research. Specifically, the age categories were adopted from Reutzel and Mohr's (2015) 50 Years of Reading Research Quarterly (1965: Looking Back, Moving Forward. The research foci categories were developed based on the five critical areas outlined by National Reading Panel (NRP)'s report (2000) and also from existing content analyses on reading and literacy research (e.g. Bauer & Theado, 2014;Morrison et al., 2011;Parsons et al., 2016;Schumm, Lewis-Spector, Price, & Doorn, 2014;Reutzel & Mohr, 2015). It should be noted that the coding of information in part two of the analysis (e.g. participant age, special education, and research foci) was not mutually exclusive. For instance, a project may have had multiple research foci such as fluency, vocabulary, and reading comprehension, and thus could not be contained into one category. In the same vein, most proposals included participants from more than one age category, and so were coded accordingly.
We did not institute a limit on how many times participant age groups, special populations, or research foci could be coded from a single proposal abstract; we coded as long as each of the components was mentioned in the abstract, with the expectation that this would provide us with a more comprehensive understanding about the NCER research grants that have been distributed over the past 13 years.

Coders and inter-rater reliability check
Ten graduate research assistants participated in coding the data. After multiple comprehensive discussions about the coding categories and processes, each graduate assistant coded independently. Upon completion, an inter-rater reliability analysis was conducted by re-coding 20% of the proposal abstracts. Responses from the two coders were used to calculate the percentage of agreement. High inter-rater agreement (0.90) was obtained. In some of the analyses (e.g. 2a, 2b, 3b, and 3c), chi-square tests of independence were calculated to determine associations between variables. Specifically, the analyses of 2a, 2b, and 3b, focusing on the yearly trends of the number of projects, were divided into two periods to reflect the shifts in administration in the US Government: Period I (2004-2009) and Period II (2010. When significant associations were found between the number of projects and the periods, cross-tabulation was utilized to determine whether the trends were increasing or decreasing. For the analysis of 3c, a chi-square test was used to detect relationships between the number of projects with different research foci and the population groups.

Results
The purpose of this study was twofold: (a) to examine how NCER grants have supported research on various areas of reading development and instruction and (b) to identify the major trends and issues in reading development research projects funded by NCER over the past 13 years. We begin by presenting the general characteristics of the funded projects, after which we report on the funding awarded for different participant groups. Finally, we analyze the frequency of each major research topic within NCER-funded projects.

The number of projects and funding amount
Research question 1 revolved around the general characteristics of NCER grants. As shown in Table 1, over the past 13 years NCER has allocated approximately $397 million to 158 reading research projects; taking into consideration the rate of inflation during this time, the amount translates to approximately $448 million in 2016 currency value (CPI inflation calculator, 2016). As noted earlier, this amount does not include funding for research on learners with mental or physical learning disabilities.
Using the basic information identified in the first part of the coding scheme, we analyzed the number of projects by year and annual funding amount (see Table 1). The largest amount of funding (approx. $160 million in 2016 currency value) was distributed across 23 projects in 2010, and the smallest amount of funding (approx. $8 million in 2016 currency value) was awarded to five projects in 2013. It is interesting to note that around 20 times as much funding was invested in research projects in 2010 as in 2013. Excluding two years where the amount of monies awarded varied considerably, the funding amount has remained relatively consistent over the past 13 years, averaging about $25 million annually (SD: 6 million; Range: approx. $13 million-$30 million, in 2016 currency value). Table 2 shows the projects funded by each program. Over half of the reading research projects (53.2%; n = 84) were funded by the Reading and Writing program. Cognition and Student Learning (11.4%, n = 18) and English Learners (11.4%, n = 18) also provided a significant amount of monies, followed by Effective Teachers and Effective Teaching (10.1%, n = 16) and Early Learning Programs and Policies (8.2%, n = 13). Only six reading projects (3.8%) were funded by the Education Technology program. Notably, in 2010, there was a one-year special program labeled the Reading for Understanding Research Initiative under which six research projects were funded.

Programs and research goals
Findings from this analysis mirror those on the number of awarded grants, with only one exception; in 2010, approximately $113 million was provided to the special program, which makes up around 28.5% of the total funding over the past 13 years.
When it comes to research goals (see Table 3), 39.2% (n = 62) of the projects were aimed at Development and Innovation, which supports the development of new interventions and the refinement of existing interventions ("IES", 2016), and 23.4% (n = 37) pursued the Efficacy and Replication goal, which supports the evaluation of fully developed education interventions ("IES", 2016). The Measurement (15.8%, n = 25) and Exploration goals (10.8%, n = 17) took third and fourth places, respectively. When looking at the amount awarded for each goal, the most funded was Multiple Goals (approx. $113 million, 28.5%), which applies only to the Reading for Understanding Research Initiative. This was followed by Efficacy and Replication (approx. $95 million, 23.9%) and Development and Innovation (approx. $94 million, 23.7%). A relatively small amount of funding was allocated to projects with Measurement (approx. $40 million, 10.1%) and Exploration goals (approx. $20 million, 5.0%), and only one reading research project was identified under Effectiveness, a new goal in 2016 (approx. $3 million, 0.8%). According to NCER, this new research goal aims to evaluate the effectiveness of a fully developed intervention using existing evidence of efficacy.

Age groups
Research question 2 is concerned with the characteristics of participants in funded projects, paying specific attention to trends in this area. Our analysis showed that the most frequently studied age group was Early Childhood (45.6%, n = 72), followed by Middle Childhood (41.8%, n = 66), and then Adolescence (29.1%, n = 46). Pre-K or Below (19%, n = 30) was the least funded age group. Table 4 shows the yearly change in the number of projects by participant age. It is evident that research projects on learners in early and middle childhood are consistently funded each year, while funding for research with learners in Pre-K or Below is awarded much less frequently. However, it was noted that the frequency of research on learners in Pre-K or Below has increased in recent years.
Special populations, including ELL and low-SES students, were coded by category only when the project explicitly stated the inclusion of these special populations in the abstract. Fifty-seven out of the 158 projects either exclusively studied ELLs (n = 30) or included both ELLs and native English speakers as their participants (n = 27), and 48 out of 158 included participants identified as low SES. Seventy-three studies either did not include or did not explicitly specify whether special populations were used in their study.

Special populations
To identify the trends in research projects that included special populations, the proportion of studies with ELL or low-SES students was calculated by year against the total number of projects funded by NCER. Figures 1 and 2 show that the proportion of research projects including either ELL or  A chi-square test of independence was conducted to compare the proportion of research projects including either ELLs or low-SES students. The 13-year range was once again divided by yearly periods: Period I (2004-2009) and Period II (2010. Results found no significant associations between yearly periods and the number of projects, ELLs: χ 2 (1, N = 158) = 0.023, p < 0.880, and Low SES: χ 2 (1, N = 158) = 2.785, p < 0.095. There were also no significant differences in the proportion of research projects with ELLs and students from low-SES backgrounds in either the first or second periods.

Research goals by populations
The research goals for each project were analyzed by participants. Table 5 presents the number of projects with different research goals by participant age group. Development and Innovation was the predominant research goal for all age groups, with a similar amount of early childhood, middle childhood, and adolescent student (n = 23, n = 24, and n = 20 respectively) studies identified. The Efficacy and Replication goal for early childhood and middle childhood learners was relatively well funded, however, both Pre-K or Below and Adolescence were the least funded age groups across all research goals, particularly within Efficacy and Replication and Exploration goals. Table 6 shows the analysis of Research Goals by special populations: ELLs and Low-SES students. Again, Development and Innovation was the research goal that most frequently included both ELLs (n = 25) and low-SES students as participants (n = 30). Low-SES students were researched the least frequently under the Exploration goal, followed by Measurement; ELL students were studied the least frequently under the Exploration goal.

Research foci
Research question 3 examined trends in the research foci of funded projects. Based on existing content analyses of reading research, we identified 10 major reading research foci to code, including Beginning Reading, Word Recognition, Vocabulary, Comprehension, and Technology (see Table 7). Similar to the population coding categories, none of these research focus categories were mutually exclusive; a proposal could be coded more than once if it had more than one research foci. It should  be noted that when we calculated the funding amount for a particular focus, we divided the award amount for a research project by the number of research foci coded. This step prevented us from over-representing the funding of studies that have multiple research foci. Table 7 presents our analysis of the number and amount of funding by major reading research foci. Comprehension (n = 93) was the most frequently studied research foci and received the largest amount of funding (approx. $128 million) over the past 13 years, followed by Vocabulary (n = 70, approx. $68 million). Content Literacy (approx. $51 million) was awarded the third largest amount of funding for the seventh largest number of projects. In contrast, Affection was the least funded research foci, with only eight projects funded (approx. $9 million). Notably, the average funding amount per project was highest for the Content Literacy focus.

Trends in research foci
We analyzed the yearly change in the number of projects by research foci, which is reflected in Table  8. Frequency counts revealed that Comprehension, Vocabulary, and Word Recognition were funded in at least one reading research project every year. If we look specifically at Comprehension, the largest number of projects were funded in 2010 (n = 17), followed by 2005 (n = 12), and 2004 (n = 9). On the other hand, the frequency was the highest in 2009 and 2015 (n = 10) for Vocabulary.
Chi-square tests were conducted to identify increasing or decreasing patterns by examining the relationship between the number of projects with different research foci and the yearly periods

Research foci by populations
We examined the number of projects with different research foci by participant age groups and special populations. As shown in Table 9, middle and early childhood comprehension received the most attention (n = 48, n = 47, respectively). In contrast, there were no studies focused on the following categories: Beginning Reading for Adolescence; Content Literacy for Pre-K or Below; and Affection for Pre-K or Below and Low-SES students. Chi-square tests were conducted to examine the association between research foci and populations. As can be seen in the

Discussion
The purpose of this study was to find the key characteristics of reading research projects funded by NCER over the past 13 years, and to identify the major trends and issues of those funded. In total, 158 projects were reviewed from which we identified the prevalent characteristics, trends, and issues. In what follows, we will discuss our findings and provide comments.

General characteristics
When it comes to the general characteristics of NCER grants, we will first discuss the amount of funding awarded over the past 13 years. The total amount awarded to 158 reading research projects  Yamaguchi and Hall (2016), NCER and NCSER have funded 203 projects in mathematics education and 139 projects in science education from 2002 to 2013. Even though the year range is slightly different and the present study only focused on NCER, the number of research projects in mathematics and science education is comparable with that of reading research projects (n = 158).
While there have been many fluctuations in total funding amount and changes in educational policies throughout these years, both the number of funded projects and the amount of funding have remained relatively stable. The reported spike in 2010 was primarily due to a special program initiated by NCER named the Reading for Understanding Special Initiative, which supported a set of connected projects aimed at improving deep comprehension and designing new interventions and assessments across all grade levels (Douglas & Albro, 2014). Under this program, NCER awarded about $113 million to six research projects for a five-year period. When this one-time program was excluded from our analysis, the number of projects and the amount of funding in 2010 were comparable with other years.
Of the 12 programs under which the NCER provides funding, Reading and Writing was the most frequently supported. Because we used "reading" as our search keyword, the high frequency of Reading and Writing (n = 72, 50.3%) is not surprising. However, the fact that approximately half the reading projects we identified were supported by various programs such as Cognition and Student Learning, Effective Teachers and Effective Teaching, English Learners, Early Learning Programs and Policies, and Educational Technology shows that funding has been provided for the study of reading development across myriad perspectives and research contexts.
Our analyses indicate that the most frequently pursued goal for reading research projects was Development and Innovation, under which researchers develop new interventions or refine existing ones. Despite that, the funding amount was found to be the highest for the Efficacy & Replication goal, even though there are far fewer funded projects under this goal than under Development and Innovation. This finding is inconsistent with that of Makel and Plucker (2014), who determined that only 0.13% of articles in the top 100 education journals were replication studies. We posit that this might be attributed to a publication bias that privileges developmental studies and considers replication studies inferior. As Makel and Plucker (2014) pointed out, the research community perceives replication studies as lacking in originality. However, given the ever-increasing emphasis on reliable and credible educational research (Burkhardt & Schoenfeld, 2003), efficacy and replication studies deserve to be featured in educational journals and funded through research grants.
Interestingly, we found that measurement studies have received the least amount of funding over the studied period. In their content analyses of articles published in Reading Research Quarterly over the past 50 years, Reutzel and Mohr (2015) predicted that assessment will become the predominant focus of research in coming years due to the current nation-wide emphasis on the Common Core State Standards (CCSS). In accordance with this forecast, we recommend that reading researchers devote more attention to measurement studies, and that major government agencies increase funding in this area in order to address issues relevant to the administration, interpretation, and application of large-scale assessments.

Participants
In terms of studied populations, K-3 participants have been the most frequently examined in reading research projects funded by NCER. This emphasis on reading in early childhood is consistent with the widely accepted belief (Guzzetti, Anders, & Neuman, 1999) that early reading interventions are the most effective way to prevent reading difficulties or disabilities later (Menzies, Mahdavi, & Lewis, 2008). For example, dyslexia can be successfully remediated if interventions are conducted in early childhood (Sigman, Peña, Goldin, & Ribeiro, 2014).
Meanwhile, we attribute the increase in research projects with middle childhood and adolescent participants to two factors: First, as many new literacy studies have emphasized, readers must now develop technology-related literacy skills, the likes of which did not become essential until recent years. These new skills are typically taught after the primary grades, and thus the reported increased emphasis on studies with older children may correspond to the need for these new literacy skills. Second, the implementation of the CCSS may also have contributed to the increased attention on adolescent reading. The CCSS position the ability to read complex texts central to successful content area learning, and propose college and career readiness standards upon which students build throughout their schooling, culminating with Grade 12 (Zygouris-Coe, 2012, p. 36). Policy-makers expect that these standards will pave the way for overcoming the current adolescent literacy crisis identified in the National Assessment of Educational Progress and The Nation's Report Card: Grade 12 Reading and Mathematics 2009 National and Pilot State Results report (Grigg, Donahue, & Dion, 2010). This leads us to recommend that middle childhood and adolescent participants be included in more funded reading research projects in the future.
When it comes to special populations, our analysis did not reveal a noticeable increase in funded projects with low-SES or ELL students. This finding is alarming given the substantial growth of these populations in the United States over the studied period. For example, Garcia and Morales (2016) reported that while the total K-12 population grew by 3.9%, the number of ELLs went up by 14.4% between 2001 and 2002 school year and the 2011-2012 school year. Furthermore, Llosa et al. (2016) reported that ELLs make up the fastest growing student population in the United States. The percentage of children from low-income homes has also been increasing; according to the National Center for Children in Poverty (NCCP), the amount of children from low income homes increased from 37 to 44% between 2000 and 2014. Much research has demonstrated that ELL and low-SES students are the most likely to demonstrate difficulties with schooling; for example, Kieffer (2010) found that ELL and low-SES students are at an elevated risk of exhibiting reading difficulties in later school years. Therefore, future reading research needs to dedicate more attention to these special populations.
The analyses of research goals by age of participants and special populations revealed that Measurement goals were the least pursued in funded reading projects with ELLs and low-SES students. We recommend that researchers include these special populations within future measurement studies to examine existing biases associated with students' cultural and socioeconomic background. For example, by analyzing a high-stakes math assessment, Martiniello (2008) found that a test posed disproportionate difficulty for ELLs, above and beyond the actual test score difference based on cognitive ability. Likewise, a pressing issue in the field of reading is how to develop a valid method for the assessment of reading skills that take into account diverse linguistic and cultural backgrounds.

Research foci
With respect to the query about research foci, we found that Comprehension was the most frequently examined topic of the 10 major foci we reviewed. The large number of projects focused on Comprehension aligns with previous content analysis literature on published literacy articles (Morrison et al., 2011;Parsons et al., 2016). As noted, our research foci coding categories were not mutually exclusive, and so the multiple coding system we employed determined that Comprehension is the most popular dependent variable. It became apparent that most of the funded studies focused on various aspects of reading development such as Vocabulary, Word Recognition, Teacher Education, and Technology with the ultimate goal of examining how these aspects contributed to general reading comprehension.
In line with the ever-increasing amount of attention to technology use in reading instruction (Cheung & Slavin, 2011), we identified a significant amount of funding for the Technology foci. This finding is consistent with previous content analyses on articles published in peer-reviewed journals. For example, in their content analysis of nine literacy journals, Parsons et al. (2016) identified that the heightened focus on Technology is one of the prominent shifts in reading research. Reutzel and Mohr (2015) reported that a topical focus on technology has been steadily rising in articles published in Reading Research Quarterly over the past 50 years. Considering that mobile devices like smartphones and tablets have dramatically increased our access to the Internet, we suggest that reading research projects focusing on educational technology should receive more NCER funding.

Conclusion
Research is essential in the field of education. However, the dearth of funding opportunities means that educational researchers must be aware of the types of projects that are awarded grants from the government. Such information will enable researchers to guide their efforts toward topics and populations that have been funded in the past, and direct researchers to areas that require more attention. In our review of the reading education studies from 2004 to 2016 that were funded by NCER, we found that those categorized under Reading and Writing were the most frequently funded, particularly studies focusing on comprehension and including participants in grades K-3.
While the information included in this study has provided some general characteristics of the reading research funded over the past 13 years, there are several limitations to our review. First, like many other content analyses, we condensed various contexts and topics from the NRP's 2010 report and other content analyses into several broader categories, leaving us with 10 research foci. While this smaller number of research foci allowed us to gain a more comprehensive understanding of the funded research, it may not capture the nuances inherent in various reading studies. Additionally, our categorization of the research foci may have identified certain general characteristics and trends that would not be apparent had the foci been organized in a different manner.
Second, we only included research projects that examined the reading development of typically developing children, leaving out students identified as having a learning disability. To obtain a broader picture of reading research, a future analysis can include research projects with participants that have specific learning disabilities. In the same vein, analyzing research from writing or other federal funding agencies (e.g. National Science Foundation) can extend the findings of this study.
Further research is needed to examine the relevant issues within reading development research grants. Given that funded research efforts typically culminate in articles in high-impact scholarly journals, the publication of funded research is worth tracking. A bibliometric approach (see Milesi et al., 2014) can be used to examine how many journal articles and in which journal federally funded research is most frequently published, providing a quantitative evaluation of the impact of federal funding on reading research.

Funding
The open access publishing fees for this article have been covered by the Texas A&M University Open Access to Knowledge Fund (OAKFund), supported by the University Libraries and the Office of the Vice President for Research.