A Randomised Control Trial Evaluation of a Literacy After-School Programme for Struggling Beginning Readers

to this body of evidence through a randomised trial ( n =464) of an after-school literacy programme ( Doodle Den ) for struggling beginning readers in a deprived area of Dublin, Ireland. The programme based on a balanced literacy framework, was delivered in group settings, and had a strong focus on staff development. Doodle Den was found to improve the children’s overall literacy ( d =+0.17), teacher assessments of their literacy ability ( d =+0.28) and to positively impact on problem behaviours in regular school class ( d = (cid:2) 0.18). The implications for the development of after-school programmes are discussed. Elsevier

supported as a means to maximise learning among all children in the early stage of learning to read (Center, 2005;Cowen, 2003;Pressley, 2006;Pressley, Wharton-McDonald, & Hampston, 2006). This programme, adopting a balanced literacy framework, was targeted at struggling beginning readers from an area of particular socio-economic disadvantage and the research design employs the random assignment of pupils to the after-school intervention (Doodle Den) or control group (business as usual).
The study tested a number of research hypotheses. The primary hypothesis was that the Doodle Den after-school programme would lead to moderate gains in the children's overall literacy in comparison to their control group peers. Other secondary hypotheses tested were related to the children's attendance at school and their concentration and behaviour within the regular school classroom. Exploratory analyses examined differential response to intervention according to year cohort, gender, family affluence, the child's ethnicity and the number of sessions attended.

Literature review
To date, the research from major educational review bodies, systematic reviews and meta-analyses in relation to what works with literacy interventions on beginning readers' shows the evidence is both rich and limited. There are many studies with strong evidence of the effectiveness of literacy programmes delivered in the normal school context, but less evidence in relation to after-school programmes focused on academic outcomes. Before examining the literature on after-school programmes some of the wider evidence on literacy interventions will be considered. Slavin, Lake, Cheung, and Davis (2009), provide an extensive review of experimental evaluations for beginning readers. They categorise literacy interventions into four broad typologies: reading curricula, instructional technology, instructional process programmes and mixed methods programmes that combine a new curriculum with professional training.
From the existing evidence that met their criteria, Slavin, Lake, Cheung, et al. (2009) found that Instructional Process Programmes showed the largest impact with an average effect size of +0.37, although there was an indication that some of these programmes were more effective than others. Cooperative learning and phonics-focused professional development showed particular promise with an average effect size of +0.46 and +0.43, respectively. Programmes that combine a new curriculum with professional training were found to have an average effect size of +0.29. Reading curricula and instructional technology programmes appear less effective, but on average, showed positive but less promising results, with average effect sizes of +0.12 and +0.09, respectively. Slavin, Lake, Cheung, et al. (2009) review concluded that successful programmes had a number of common elements, these included: extensive professional development and follow-up on specific teaching methods for teachers; co-operative learning at their core, with children working together on structured activities; a strong focus on teaching phonics and phonemic awareness and well-developed programmes that integrate curriculum, pedagogy and extensive professional development.
In a Best Evidence Encyclopaedia review focusing upon struggling readers at the early elementary stage Slavin, Lake, Davis, and Madden (2009) concluded that, regardless of programme type, a strong emphasis upon phonics was evident in the most successful programmes, and one to one programmes appear particularly effective especially if delivered by a teacher rather than a paraprofessional or volunteer. These programmes, however, tend to be expensive to deliver and while the evidence suggests they are more effective than small group programmes, the review found that instructional programmes with a focus on cooperative learning and phonics could provide average effect sizes among the lowest performing children that were similar in magnitude to one to one tutoring (average effect size +0.56). These syntheses provide clear evidence of the types of programmes and the factors associated with improvements in the literacy outcomes of young struggling readers when delivered in the context of the normal school day. However, the next section reviews the evidence in relation to afterschool programmes that focus on academic outcomes where the evidence to date presents a much more mixed picture.
After-school programmes have expanded in recent years, particularly in the USA where they have been supported by federal funding for over a decade. Initially, these were established to address concerns about keeping children safe and out of trouble given changing patterns of labour force participation. After-school programme goals can be wide and varied, from providing a safe environment and recreational activities, to homework clubs, or more structured programmes that have typically focused on behavioural or academic outcomes. In recent times, in the USA, they have been adopted as a means of raising achievement and closing the attainment gap promoted through the requirements of the No Child Left Behind legislation.
A What Works Clearing House practice review of out-of-school time programmes focusing on academic outcomes identified a number of after-school programmes that met their standards with or without reservations (Beckett et al., 2009). Five studies showed positive impact, although sample sizes in these studies were small ranging from 48 to 84 children. The larger studies included in their review were located among those that showed either mixed effects (1 programme) or no discernible effects on academic outcomes (6 programmes). One of the largest studies included in their review was the national evaluations of the 21st Century Community Learning Centers in the USA, federally funded after-school programmes, which found no impact on average attainment such as reading and mathematics (Dynarski et al., 2003(Dynarski et al., , 2004. The evaluation, however, highlighted that the opportunities presented for instruction in these core subject areas was often limited within many of the programmes. A more recent systematic review on the topic also reported no effects on reading outcomes for after-school programmes, but this was limited to only 5 randomised control trial evaluations that met their rigorous inclusion criteria, which also included reading outcomes (Zief, Lauer, & Maynard, 2006).
More promising evidence is provided through another meta-analytic review of after-school programmes that focused upon interventions with academic outcomes (reading and math) for children deemed at risk (Lauer et al., 2006). This analysis was not restricted to randomised trials but also included quasi-experimental designs where a control or comparison group was evident. Of the thirty studies with reading outcomes, effect sizes varied considerably, but an average effect size of +0.05 or +0.13 was reported depending on whether it was measured as a fixed effects or random effects model. Overall, they found a tendency towards positive effects for after-school programmes aimed at improved reading outcomes for at risk students.
The mixed findings of the existing reviews on the academic benefits of after-school programmes may be related to the different focus of the reviews and the limited number of rigorous evaluations that have been conducted to date. Problems in assessing the evidence include: the wide heterogeneity of programme activity and structure; varying target audiences; differing programme quality and goals; the varying ages of participants and the extent of programme duration. These may therefore under-estimate the benefits of good quality well-structured programmes that focus on academic outcomes, especially for struggling beginning readers. Nevertheless, specific attempts to assess the use of enhanced well-structured after-school programmes to improve reading outcomes have also led to disappointing results (Black, Somers, Doolittle, Unterman, & Grossman, 2009). Apsler (2009) highlighted some of the problems or common issues that tended to arise in evaluation studies of after-school programmes. These included a lack of appropriate control groups and selection bias in terms of both research design and programme participation. Furthermore, high attrition is seen as a particular problem for evaluation designs due to the voluntary nature of participation in after-school programmes, especially among those serving older children or youths (Scott-Little, Hamann, & Jurs, 2002).
While there are many advocates for the use of after-school programmes the evidence base remains weak and recent best practice guides in after-school programming have concluded the level of research was not sufficient to provide conclusive recommendations (e.g., Beckett et al., 2009). However, a number of tentative features have been highlighted as potentially important components of successful after-school programmes aimed at improving academic outcomes. Some of the emerging features of quality after school programmes include the importance of aligning the programme to the school day (Beckett et al., 2009;Fashola, 2002); maximising student attendance (Beckett et al., 2009;Fashola, 2002;Scott-Little et al., 2002); adapting instruction to individual and small group needs and the provision of engaging learning experiences (Beckett et al., 2009); programme structure; one-to-one tutoring and the importance of well qualified and trained staff (Fashola, 2002).
Other reviews have suggested that effects may differ according to pupil characteristics and the largest positive programme effects for reading were among those that focused on children at risk among those in the lower elementary grades (Lauer et al., 2006;Scott-Little et al., 2002).
The previous literature provides a number of main conclusions. Firstly, a general consensus has emerged within the English speaking world that a balanced literacy framework is currently considered a promising approach in the early stage of learning to read. Secondly, although they are resource intensive one to one programmes show particular promise. Thirdly, mixed method literacy programmes (curriculum plus staff development) that include phonics are a promising alternative way to develop literacy for beginning readers. Finally, despite the major growth in after-school provision, these services remain an under investigated area of educational development, and there is little rigorous evidence (RCTs) to support the use of structured after-school programmes as a means of improving academic outcomes among struggling beginning readers. It is in the context of these wider debates and the existing evidence base that the current evaluation is presented.

The Doodle Den after-school programme
Doodle Den was a newly developed manualised after-school programme that was devised in conjunction with literacy experts at a local university (O'Rourke, Kennedy, & Axford, 2008). It aimed to target struggling beginning readers (5 and 6 year olds, Irish: Senior Infants class) in an area of significant socio-economic disadvantage and involved children attending three after-school sessions per week, each lasting one and a half hours. The programme operated throughout the normal school year, over a 36 week period and was delivered by two staff -a qualified teacher and qualified youth worker or childcare professional -and operated in seven different settings with groups of up to 15 children drawn from 8 schools.
The programme had a very structured approach in that each session began with a snack and sign-in routine, was followed by various aspects of literacy teaching and activities and concluded with a 'fun' element (art, P.E, drama or music) based on a literacy theme. Each element of the programme was given a specified time, within each one and half-hour session.
Adopting a balanced literacy framework the programme aimed to improve children's literacy through targeting the following literacy domains: writing, text comprehension, phonics, sight vocabulary, independent reading and fluency. The concept of a 'balanced literacy framework' is varied and contested with regard to what constitutes a balanced framework and whether or not it should be promoted as the most effective way to teach literacy (Adams, 1990;Moats, 2000). There is no standard definition of a 'balanced literacy approach', rather, studies tend to outline the essential or necessary components and some enter into debates in relation to specific components. Doodle Den can be said to encompass a balanced literacy framework in the sense that the main elements, within a given session, included a combination of various modalities of literacy instruction: phonics 'mini-lessons', sight vocabulary, shared reading, independent reading, shared writing, independent writing and comprehension. It therefore combined a skills based approach with an immersion in a literacy rich environment.
Activities were designed to be fun and engaging for the children in order to try and ensure it did not seem simply as an extension to the normal school day. It also drew on the concept of emergent literacy in terms of the age group targeted where literacy is seen to begin to emerge naturally for children at approximately six years old (Clay, 1966).
In addition to the children's sessions, there was also a family component, whereby parents were encouraged to participate in a range of activities to include sitting in on child sessions and shared reading activities designed to promote wider family literacy. A joint family activity was organised by the service providers each term and in total there were three family and six parental sessions.
There was also a strong emphasis on staff development among those delivering the programme, with a minimum of 20 h training a year, monthly communities of practice meetings. 'Communities of Practice' meetings (COPs) were outlined as a network formed to create a community of practice (Liberman, 2006). This particular community was supported by a 'Quality and Services Officer' and a 'Quality Specialist' and was attended by facilitators, service providers and other interested parties. They were outlined as ''key support for service providers'' and that they may include key speakers, case study presentations or facilitated sessions on key issues of delivery (O'Rourke et al., 2008, p. 69). The objectives outlined related to supporting fidelity, providing technical assistance in delivery, a space for reflection, consideration and the sharing of learning as well as a place to identify and respond to training needs and collectively identify solutions to issues.
The stated aims of the Doodle Den programme, according to the manual, were to achieve moderate improvements in the children's literacy (O'Rourke et al., 2008), with specific outcomes: improvements in the children's broader engagement in learning outside the school through improved literacy, increasing confidence and improved home environment relating to literacy; improvements in children's regular school attendance and enhancing relationships between the child and their family and peers. For example, through increasing parent/carers involvement in supporting their child's literacy development and increasing family use of library services.

Methods
The trial was designed as an individually randomised trial within multiple schools utilising a three year rolling cohort design. In other words, it followed three different cohorts of children over three successive school years. The evaluation began in September of the 2008/09 school year and the final cohort was post-tested in June 2011. Children completed a pretest before the start of the programme in September and the post-test assessment at the end of the programme in the following June.
It was proposed that 210 children would be referred to the programme each year for three years, giving a maximum proposed total sample of 630 children. With the rolling cohort design particular care needed to be taken over potential adverse effects on the evaluation. Therefore the main outcome results and comparison of pre-and post-test scores between the control and intervention were not analysed until the final cohort had completed the programme. Releasing interim results may have had undesirable and unintentional effects on the delivery of the programme and/or undermine the validity of the evaluation. In addition, any interim outcomes would have only been tentative, but could also have been misleading.

Participants
The main participants in the study were five and six year old children living and attending 8 different primary schools in an area of Dublin designated as an area of particular socio-economic disadvantage. This was deemed as an appropriate age as the children were: ''old enough to have started to learn to read but young enough not to have become disengaged from reading'' (O'Rourke et al., 2008, p. 53) Teachers in eight schools were asked to refer children to the Doodle Den programme on the basis of their needs as struggling beginning readers. Control group children received their normal school classes alongside the Doodle Den children in regular school, but were also offered the opportunity to attend a week long literacy based summer scheme once the Doodle Den programme and the evaluation was completed. Table 1 summarises the main characteristics of the sample and is broken down by intervention and control group in relation to gender, special educational need (SEN), ethnicity and cohort. The children on average at pre-test were approximately five and a half years old (control M = 5.64, SD = 0.36; intervention M = 5.62, SD = 0.37). Fig. 1 shows that there was some attrition during the various stages of testing. The major reason for this was that children were absent from school on the day of testing or left the school after referral was made. Multiple test dates would have been desirable although this was not possible due to the funders concerns about overburdening schools. However, the pattern of attrition was similar in both the intervention and control groups with similar numbers leaving the study at the various stages.
The final numbers in the analysis (76 per cent intervention and 73 per cent control group) were also similar. In terms of teacher assessments a similar response rate was obtained for those completing both pre and post-test (79 per cent intervention n = 245; and 73 per cent control n = 226).
The previous literature on literacy interventions suggested as an individually randomised trial that effect sizes may be in the range of 0.15-0.30. A priori power calculations suggested a sample of 500 would be sufficient to detect a minimum effect size of 0.15, assuming a pre-post-test correlation of 0.80 (R 2 = 0.64). The initial child sample for the study was 623 and after attrition the sample still remained at an acceptable level (n = 464). This should be sufficient to detect significant effect sizes as low as 0.16. [ ( F i g . _ 1 ) T D $ F I G ]

Assessment procedures
The trial comprised child, school and teacher measures as well as parental measures. At both pre-and post-test children from intervention and control groups, within the same school class, were tested together during a regular school day. The child testing was conducted in a group setting and was overseen by members of the research team and/or trained fieldworkers. All fieldworkers were police checked and given training in the assessment procedures prior to undertaking testing. Teacher child assessments were undertaken by the child's regular class teacher and involved completing a questionnaire for each child in their class who were allocated to either the intervention or control group. While it was impossible to blind participants including teachers, the fieldworkers were unaware of whether children were treatment or control at the time of testing. Parental measures were posted to the children's addresses and returned to the research team by post. Parental non-responses were also followed up by telephone.

Child measures
The children's overall reading ability was measured through the Drumcondra Primary Reading Test, a standardised measure of reading ability widely used in schools in Ireland to monitor educational progress and normed on 1381 pupils of a similar age (Sheil, 2008). There are four separate sub-scales to the test: word recognition (15 items); picture recognition (5 items); sentence structure (5 items) and word choice (5 items). The overall Cronbach's alpha for the test at pre-test was 0.79 and for the individual subscales word recognition (0.78), picture recognition (0.53), sentence structure (0.70) and word choice (0.67). Due to lack of reliability of the picture recognition sub-scale this was not analysed as a separate outcome, but it was included in the overall Drumcondra score.

School and teacher measures
In addition to the direct measures of the children's literacy, a number of teacher measures were collected. This included a teacher assessment of the children's literacy abilities using an adapted version of a nationally developed test (8 items) in Ireland (Eivers, Shiel, Perkins, & Cosgrove, 2005). This displayed very good reliability (alpha = 0.94). Teachers were also asked to assess each child's concentration and behaviour in regular school classes through the (14 item) attention deficit hyperactivity disorder rating scale (DuPaul, 1991). This also displayed high reliability (alpha = 0.94).
The children's regular school attendance record was also collected directly from the eight schools by the research team; this was only collected at post-test. As a measure of 'dosage' the number of Doodle Den sessions attended by each intervention child was also collected.

Parental measures
A number of parental measures were collected designed to measure parental perceptions and family literacy activity. However, despite significant and on-going attempts to maintain parental response rates, including reminder letters, questionnaire re-issues and offers of telephone completion the parental response rate remained below an acceptable level (32 per cent completing both pre-and post-test) and the results are not reported in detail here. This highlights the serious difficulties in securing high response rates from parents in an area of socio-economic disadvantage, a problem that may be particularly acute in Ireland. As well as a number of other potential barriers, some parents may have significant literacy difficulties themselves although many still did not avail of the offer of telephone completion. For transparency the reliability and effect sizes are reported below. The Cronbach's alpha for the parental measures were: parental reported child reading (alpha = 0.62), parental reported library activity (alpha = 0.75), parental reading attitudes (alpha = 0.65) and parental report child's literacy activities (alpha = 0.90). Two of the parental measures showed significant positive effects for the intervention children: parental reported child reading (d = 0.25, p = 0.047) and parental reported library activity (d = 0.39, p = 0.004), but parental reading attitudes (d = 0.08, p = 0.462) and parental report child's literacy activities (d = 0.18, p = 0.191) while also positive were non-significant.

Analysis
The main statistical analysis in the study was conducted on an intention to treat basis using multiple linear regression. An analysis of pre-test scores highlighted there were no significant differences between the control and intervention group on any of the outcome measures showing the two groups were still well matched following attrition and, therefore, suitable for outcome comparison (Table 2). Adjusted post-test means were also calculated for each of the groups, controlling for pre-test scores. Effect sizes were then calculated as standardised mean differences (Cohen's d). There was no clustering adjustment made to coefficients as participants had been randomised at the individual level (not school level). Further pre-specified subgroup analyses were conducted by adding interaction effects to the regression models to examine the effect of gender, ethnicity, family affluence and year cohort.

Implementation
While the focus of this article is on the main outcomes of the trial the study incorporated a qualitative process evaluation involving interviews with staff, parents, children and other key stakeholders, and observations of the sessions at each of the sites. This highlighted how the facilitators were encouraged and worked hard to maintain fidelity, but due to the number of planned activities in a given session these sometimes took longer than expected and on occasions some activities were missed, often the more fun element at the end of the day. Differentiation was another issue and can be one that poses difficulties for manualised approaches when there is a considerable spread of abilities within a given group. While facilitators developed approaches to deal with the problem of differentiation this could lead to some minor tensions with strict adherence to the manual.
Children who received the intervention attended an average of 63 sessions (SD = 21.29), and the minimum number of sessions attended by a child was one and the maximum was 88. The number of sessions a child could have attended varied according to delivery setting and ranged between 67 and 89. Overall levels of participation in the programme were high with 75 per cent of the children allocated to Doodle Den attending over 60 sessions and only 10 per cent attending fewer than 30. Table 3 presents the regression analysis for all the child and teacher measures. The results show that after controlling for pre-test scores, intervention children scored significantly more positively than control group children on all of the measures with the exception of the children's regular attendance at school. Table 4 summarises these main primary effects reporting: the adjusted post-test score (calculated from the unstandardised betas from the regression) with standard deviations; the effect size (Cohen's d) difference between the control and intervention group for each of the outcomes (Cohen's d) and whether the differences are statistically significant (i.e., p < 0.05). Statistically significant effects are indicated in bold.

Main analysis
As can be seen from Table 4, in terms of the main primary literacy outcomes, those children who participated in Doodle Den scored significantly higher than the control group children in their overall measured literacy ability (d = +0.17) as well as the separate teacher rating of the children's literacy ability (d = +0.28). The overall measure of the children's literacy contained a number of sub-scales indented in Table 4 (word recognition, sentence structure and word choice), all of which were also statistically significant. The results highlight a further significant effect for the children who attended Doodle Den; with a reduction in teacher reported problems in terms of concentration and behaviour in regular school class (d = À0.18) as measured through the ADHD rating scale (DuPaul, 1991). There was no reliable evidence that Doodle Den had an impact on the children's general attendance at regular school. To make these effect sizes easier to interpret they can accurately be translated into an average percentile gain. This shows that on average children who attended Doodle Den had a 7 percentile point gain in their overall literacy ability compared to the control group. When teacher ratings are considered this gain increased to 11 percentile points. The average percentile gains for word recognition, sentence structure and word choice were 7, 12, and 10, respectively.

Exploratory analysis
A number of pre-specified subgroup analyses were undertaken to see whether Doodle Den worked differently for: boys and girls; different year cohorts; family affluence/poverty (as measured by the family affluence scale II, see Schnohr et al., 2008) and ethnicity. In addition to this, analyses were conducted to explore whether the number of Doodle Den sessions that were actually attended (i.e., a higher level of exposure to the programme) was related to better outcomes for the intervention children. These were calculated by adding the appropriate interaction effects into the regression models. There was no evidence in relation to either child or teacher outcomes that the programme worked differently according to year cohort or family affluence, although some significant differences were found in relation to gender, ethnicity and the amount of the Doodle Den programme the children received and these are reported and discussed in more detail below.

Gender.
There was no evidence that the programme worked differently for boys and girls according to the children's or teacher measures of literacy ability. However, in relation to one of the other teacher measures a gender difference was observed in terms of teachers reported concentration and behaviour in class (p = 0.009). For transparency the full results of the regressions are shown in Table 5.
In order to explore this significant difference further, we can use the regression model to calculate the predicted posttest scores controlling for the pre-test score for each of the groups to examine the nature of this difference. This revealed that the predicted post-test scores were similar between the control group of girls and the girls who attended Doodle Den (M = 0.65, SD = 0.72 and M = 0.62, SD = 0.54, respectively). However, a notable difference was apparent between the boys (control: M = 0.87, SD = 0.72; intervention: M = 0.63, SD = 0.72). Intervention group boys were significantly less likely to display a lack of concentration and poor behaviour in class, as rated by teachers at post-test (controlling for their pre-test   scores), when compared to the boys who did not attend Doodle Den. In fact, the rating of the Doodle Den boys at post-test was similar to both groups of girls (control and intervention), with the control group boys exhibiting the most problem behaviours in regular school classes. In other words, there were more problem behaviours in regular class reported by teachers among the boys compared to the girls, but the Doodle Den boys' behaviour appears on average to have improved to a level similar to the girls.

Ethnicity
A similar sub-group analysis was run in relation to the child's ethnicity (Table 6). Overall, the sample was predominatly comprised of White Irish children although 17 per cent of the children were recorded by teachers as having another ethnic background. While these children came from a wide variety of nationalities the most common were African (in particular Nigerian), Irish Travellers and Eastern European migrants. Of the children that teachers indicated were from an ethnic minority background around two-fifths (39%) were reported as not speaking English as a first language at home. The analysis compared the sample of White Irish children to the sample of other ethnicities. Once again this revealed that the programme did not work differently in relation to the literacy measures for children of different ethnic origins. However, again a significant difference was observed in relation to teacher reported concentration and behaviour in regular school class.
The predicted post-test scores suggested that children who did not attend Doodle Den from a non-ethnic minority background (White Irish children) were more likely to have higher levels of these problem behaviours (M = 0.77, SD = 0.88), compared to both their respective intervention group (M = 0.59, SD = 0.66), and the two groups of minority ethnic children (control: M = 0.57, SD = 0.71 and intervention: M = 0.60, SD = 0.67) whose scores were similar. In other words, ethnic minority children in general seemed less likely to display these problem behaviours in regular school class and the programme appeared to reduce these behaviours among the non-minority ethnic children into line with the scores of the minority ethnic children.

Number of Doodle Den sessions attended.
The final exploratory analysis was restricted to children in the intervention group and was designed to test whether the extent of exposure to the Doodle Den programme had an impact on outcomes. Programme facilitators kept a weekly register  of children who attended the programme. Children who received the intervention attended an average of 63 sessions (SD = 21.29), which equates to an average contact time of 95 h additional literacy activities over the course of the school year. No evidence was found that greater attendance at Doodle Den impacted on sentence structure, word choice, word recognition, or concentration and behaviour in class. However, the analysis found evidence that the greater the number of sessions attended by the children, the greater the gains in overall literacy ability (p = 0.033), attendance at school (p = < 0.0005) and the teacher rating of the child's literacy skills (p = 0.027) (Table 7).

Discussion and conclusion
Overall, the Doodle Den after-school programme was found to lead to moderate improvements in children's literacy ability assessed through standardised direct measures of the children's literacy, and triangulated through detailed teacher ratings of their literacy ability. Not only did the programme improve children's overall literacy ability, but there was also some evidence to suggest that it had a positive impact on improving concentration and reducing problem behaviours in regular school, among the non-ethnic minority boys in particular.
These results contribute to the field of after-school programming in a number of ways. Firstly, the results add to the dearth of randomised trial evaluations internationally of academically orientated after-school programmes that are delivered to young children at risk of academic failure. While some large scale evaluations in the USA have raised questions about the value of after-school programmes in improving academic outcomes (Dynarski et al., 2003(Dynarski et al., , 2004, these have failed to account for the heterogeneity of programmes, and the current findings add to a limited but wider body of evidence that intensive well-structured programmes can be effective (Lauer et al., 2006).
Secondly, while the previous evidence on literacy programmes suggests that one-to-one tutoring is the most effective intervention for struggling beginning readers, the evidence from the current study suggests that group programmes can be effective in the after-school context. Although the effect sizes achieved were relatively modest, one-to-one programmes can be expensive to deliver and may not be well suited to the after-school context. In this respect consideration needs to be given in relation to the extent of reach, the programme setting and the costs and benefits of one-to-one programmes within a given set of resources. Doodle Den ran throughout the whole school year and provided additional instruction and childcare for the pupils attending at a cost of s1656 per child/annum.
Thirdly, while the focus of the programme was primarily on the children's literacy development, the Doodle Den children appeared to derive some additional benefits with a noted improvement in the children's concentration and reduced problem behaviours in their regular school classes. While the finding in relation to boys, as part of a sub-group analysis, should not be overemphasised it does appear worthy of further research. This comorbidity between literacy and behavioural problems in young children has been previously highlighted through correlation studies (Hinshaw, 1992;Miles & Stipek, 2006), and the current trial suggests it may be possible to effect positive change in classroom behaviour through a literacy intervention.
These findings lead to a larger question as to why Doodle Den achieved positive impact when other similar, recent attempts to assess enhanced academically orientated after-school programmes through randomised trials have found null results (Black et al., 2009). While the precise reasons remain unclear, and we cannot draw too many conclusions from a single trial, one potential avenue to be considered by future research may be the limited age range of the participants in the current study. The trial of enhanced academic programmes in the USA catered for a wide age range of participants spanning a total four school year cohorts. In this respect, even though students were grouped within the programme according to assessed reading level, it may be more difficult to design a manualised programme that caters and differentiates over such a wide range of potential literacy abilities. The Doodle Den programme, on the other hand, was targeted at a single year cohort of early struggling readers and was designed specifically for the abilities of this age group.
Even though the present study has demonstrated the overall effectiveness of Doodle Den, and raised some future directions for research, there remain a number of unanswered questions about the pedagogical practices driving these effects. In other words, the current evaluation looked at the impact of the programme as a whole and we cannot report on the effectiveness of each of the component parts as they were not independently considered. The research, however, builds upon and contributes to a number of prior tentative findings from previous reviews of successful programme characteristics: these include the age of the children; their family background; high programme attendance; well qualified staff with a strong focus on staff development and a well-structured programme aligned to the normal school day (Beckett et al., 2009;Fashola, 2002;Lauer et al., 2006;Scott-Little et al., 2002). Lauer et al.'s (2006) moderator analysis suggested that the most effective programmes appear to be among at risk students in the early elementary grades; and this not only underscores the importance of early intervention, but is also likely to be a contributory factor in the positive results reported here. Voluntary participation has also been identified as a particular problem for both after-school programmes and rigorous evaluation designs, but again, this was less of a problem in the current study due to the young age of the participants. Nevertheless the exploratory analysis has highlighted the importance of time spent exposed to the programme in order to increase effects. In other words the children who attended most frequently appeared to gain the most benefits.
There are several limitations to the study which must also be considered. Firstly, despite significant efforts to maintain parental response rates, levels remained below an acceptable level and we cannot confidently report on the impact of the programme on family literacy practices which were also pointing in a positive direction. Secondly, although the largest effect sizes were observed through teacher reports and these are supported by the direct measures of the children's literacy, the teachers were not blind to who was receiving Doodle Den and who was not. While the overall pattern of results does not suggest any bias in teacher reporting, ideally teachers would have been blind to who was receiving the intervention. However, this was not feasible in the current study. Further limitations concern the measurements used in the study and while these were carefully chosen to be age appropriate and to capture a range of literacy outcomes and other pertinent measures, there is always the question as to whether other effects have remained unmeasured. However, all the measures were agreed with stakeholders prior to testing and the reliability of measures was also monitored across the three cohorts.
To conclude, while the programme has demonstrated improvements in the children's literacy at the end of the programme, it is unclear whether these benefits will be maintained over time without further intervention. While many evaluation studies lack longer-term follow-up, some studies have shown that the initial observed effects of literacy programmes may decline sharply over time (Baenen, Bernhole, Dulaney, & Banks, 1997;Hurry & Sylva, 2007), while others have noted enduring benefits over many years (Borman et al., 2007;Ross, Smith, & Cassey, 1995;Slavin, Madden, Dolan, & Wasik, 1993). There are, however, plans to continue to longitudinally monitor all the child participants included in the current study up to three years after the programme, but at the time of writing it is still not known if the observed improvements have been maintained, enhanced or have actually declined over time relative to their control group peers.