Determining the Efficacy of a Community-based Early Intervention Program Adapted for Middle School

Middle schools are often faced with behaviorally disruptive students who concurrently struggle with their own academic problems as well as interfere with other students’ learning and functioning. Often times schools do not have the resources to provide other alternatives than school suspensions, even though suspensions often result in increasing academic difficulties and negative behavior. Previous studies have found early intervention programs delivered in a community setting, which combine self-regulation skills and art classes to be effective with at-risk youth. This study evaluated the adaptation of one such community-based program for delivery within middle schools and found promising results in problem-solving, social skills, and academic self-efficacy. School social workers advocating for the implementation of early interventions, like the PASS program, improve at –risk students’ probabilitiy of academic success. Determining the Efficacy of a Community-based Early Intervention Program Adapted for Middle School


Introduction
Disruptive, externalizing behaviors in middle school youth are often concomitant with academic problems and potential academic failure and serious recidivism. Due to the lack of resources, schools often react by suspending youth from school. Unfortunately, such action facilitates further offending, exacerbates academic difficulties, and reinforces negative behaviors. School social workers are often in the position of trying to improve students' disruptive behaviors after problems have steadily grown worse. In an effort to address the issues associated with disruptive school behaviors and avoid repeated suspensions, several Early Intervention programs have been developed [1,2].
Early intervention programs encouraging self-regulation skill development have been found particularly effective in reducing anger, offending and violent behaviors in at-risk youth [3]. In addition, self-regulation skills have been found to reduce mental health problems and improve academic self-efficacy and achievement [4][5][6]. In essence, programs utilizing self-regulatory skill development attempt to interrupt the inevitable path many adolescents travel from inappropriate classroom behavior to severe school-based discipline; eventually, becoming involved with the juvenile justice system, which may further exacerbate youths' negative attitudes and behaviors and mental health symptoms.

Self-regulation and academic performance
Although many factors affect academic achievement, there is a very strong link between self-regulation and academic skills [2,7]. Self -regulatory skills are essential in young children for school readiness and later in adolescents for overall academic success [8,9]. According to Blair [8] self-regulatory skills are separate from intelligence but equally important. Bakracevic and Licardo [10] examined students with a battery of measures and found self-regulation skills to be the best predictor of academic performance, above other variables, across all educational years in a sample of 333 students from ages 14-23. To be precise, self-regulation competencies help students block out distracters, maintain their focus, and persevere at difficult tasks and result in better performance [11]. Cleary, Platten, and Nelson [12] noted that adolescent students who were taught self-regulatory skills used more adaptive and goal-directed behaviors in the classroom to facilitate their success. Pintrich and DeGroot [11] also found that students who had high academic self-efficacy utilized self-regulation skills more frequently than other students, but academic performance was directly linked to the use of self-regulatory skills while academic self-efficacy indirectly affected academic achievement. Previously mentioned studies demonstrate the importance of implementing programs in schools that teach self-regulation because results show positive academic outcomes beneficial for at-risk students.

Self-regulation and mental health outcomes
Externalizing behavior problems such as disruptive behavior disorders and conduct disorder, as well as internalizing behavior problems such as mood disorders and anxiety in youth are often related to self-regulation deficits [13][14][15]. Youths with these mental health problems frequently lack necessary executive functions and emotional control capacities required to regulate behavior. In addition, they often exhibit problems with appraising negative emotions, and shifting attention [15]. Without these capabilities, youth often experience detrimental life consequences like substance abuse, gambling, theft, depression, suicidal ideation, and academic failure [16,17].
Interventions targeting self-regulation skills have been found to be effective in reducing mental health symptoms. Izard [18] found a decrease in externalizing and internalizing symptoms, maladaptive behavior, and negative emotion expression in preschool children after participation in a self-regulation program. Wyman [15] found improvements in behavior control, stronger social skills, decreases in disciplinary referrals, and decreases in suspensions from school. Likewise, prevention programs focusing on self-regulation skills found improved emotion regulation, increased use of problem-solving strategies, improved social skills, and fewer attributions of anger [19,20].
Overall, enhancement in self-regulation skills has shown an increase in academic achievement, improvement in mental health functioning, improved social outcomes, and increases in ability to cope with stress [8,[21][22][23][24].
A meta-analysis by Reid, Trout, and Schartz [25] found selfregulation interventions increased on-task behaviors, academic productivity and accuracy and reduced disruptive behaviors.

Effects of arts programs
Arts based programs have begun to evince promise as an intervention modality with at-risk youth. Several studies have found improved anger management, communication skills, and task completion by youth who engaged in art programs [26]. Other studies indicated an improvement in life skills, prosocial attitudes and behaviors, and an increase in overall school performance [27,28].
Student academic achievement has also been repeatedly linked with arts programs [29][30][31][32]. In fact, a recent report discussing the results of four longitudinal studies found that at-risk youth with a history of intensive art experiences showed academic achievement levels close to and in some cases exceeding levels shown by the general population [33]. McPherson and Zimmerman [34] found the arts contributed positively to school attendance and self-regulation. These results demonstrate that arts programs are a feasible solution to poor academic performance by at-risk youth and could be utilized by school social workers to improve schools' academic achievement, school attendance, and self-regulatory behaviors. Metsapelto and Pulkkinen [31] also noted self-regulation improvements specifically in the domains of persistence, concentration, and carefulness.
Arts programs have also shown success in the reduction of mental health symptoms. Jeong, Hong, Lee, and Park [35] found dance reduced symptoms of somatization, depression, anxiety, hostility, obsessivecompulsions, and psychosis in adolescents. A study regarding music intervention found improvement in adults' depression and anxiety symptoms [36]. Research in the area of arts programming is still in its infancy, as most studies are based on small samples and mainly employ qualitative methods. However, initial evidence warrants further investigation, especially for programs which combine arts programming with self-regulation skills.

Prodigy cultural arts program
The prodigy program is an after-school early intervention program for youth who have been adjudicated in the juvenile justice system, as well as non-offending, at-risk youth in the community. It is a community-based program including self-regulation skills and art classes in visual, performing, musical, media, and theatre arts. Three types of self-regulation skills are modeled, taught and practiced in the classes. These include social skills, anger management, and problemsolving skills.
The classes are taught by master artists from the community who develop positive and supportive relationships with the youths. The artists are extensively trained in learning styles, the development of lesson plans, youth development, and skill building. Trained staff monitors and observe program implementation, lesson plans, and skill delivery in the classrooms [37].
Several studies have found the Prodigy program to be effective in reducing mental health symptoms, improving academic self-efficacy, reducing recidivism, and improving overall family functioning [6,37,38]. Although the program has been successful, for those youth who have attended, there are always youth who are unable to consistently attend a program in the community.
Therefore, for this study, the prodigy program was modified to be delivered to at-risk middle Schools students who were behaviorally disruptive in school. The intent was to adapt a community program for in-school delivery to ameliorate logistical problems of families (i.e., transportation) as well as to provide an alternative to school policies which are often not helpful. The program modified for after school had the same intent of diversion from school discipline and the juvenile justice system. The program was re-titled PASS (Positive Alternative to School Suspension).

PASS program description
PASS is identical to the Prodigy program except that it is delivered on school property after school hours, 5 days per week as an alternative to school suspension. The PASS program was offered to youth who evinced disruptive or illegal behaviors in lieu of suspension from middle school. The program was voluntary and was delivered afterschool daily, for two months at two middle schools.
The art instructors and their co-instructors were trained and followed a written program manual. They were observed for fidelity to the program. Feedback was provided if they strayed from the program. Each instructor had a co-instructor in the class at all times.
This study evaluated the PASS program to determine overall efficacy. The specific research questions addressed were: 1) Are there changes in mental health symptoms, school selfefficacy or social skills for those adolescents particpating in PASS?
2) Is there a significant difference at pre-test for those adolescents particpating in PASS from those that are in a comparison group?
3) Do post-test social skills impact school self-efficacy?

Methods Procedures
A quasi experimental pre/post-test design with a comparison group was utilized to evaluate program effects on participants' mental health symptoms, academic self-efficacy and social skills. After informed consent was obtained from parents and youth, the participants completed the pretest and were placed on a list. Adolescents were randomly assigned from the list to be in the treatment or wait-list comparison group. The Treatment group attended the Pass program five days per week and after approximately two months, participants completed the posttest. Wait-list youth completed the program at a later date. A trained team of data collectors collected data by following a detailed protocol.

Sample
Ninety-five adolescents participated in the PASS study. The treatment group was comprised of 46 randomly assigned youth while the The analyzed sample was slightly unevenly divided by gender with 61% males and 39% females. Ages ranged from 11 to 16 years old with a mean of 13.8 years of age. The sample ethnicity was largely African American (52%) and Caucasian (38%) with Hispanic (10%) reported as the next largest category.
Due to the small number of adolescents dropped from the analyses, complete statistical analyses of the three youths not included in the reported results were not completed. Non-parametric examinations, however, found no differences between the dropped adolescents on any of the study or demographic variables at pre-test, providing some confidence that the non-participating adolescents did not represent outliers and that the analyzed sample was generally representative of adolescents participating in the PASS program.

Measures
Mental health: The Youth Self-Report (YSR)/6-18 has 118 items completed by the adolescents that describe specific behavioral and emotional problems (anxious/depressed, withdrawn/depressed, somatic complaints, social problems, Thought problems, attention problems, rule-breaking behavior, aggressive behavior), in youth, plus two open-ended items for reporting additional problems.
The measure was normed with a sample of 1,057 adolescents. Reliability scores ranged from a low of .67 (withdrawn/depressed) to a high of .84 (Attention Problems, Thought Problems) [39]. Both Content and Criterion Validity were assessed and found to be acceptable [39,40].
Academic self-efficacy: The adolescent's perceived ability to perform well in school was measured using a sub-scale of the YSR, entitled Academic Self-Efficacy. The YSR scale, completed by the adolescent, has eight items measuring the adolescent's perception of their ability to perform academically in five different school subjects plus an overall assessment of perceived academic ability. Reliability for this scale was .89 and both content and criterion validity was found to be acceptable [39,40].
Aggressive behavior: The Aggression Questionnaire (AQ) has 34 items assessing hostility and aggression through five separate subscales: Physical Aggression, Verbal Aggression, Anger, Hostility, and Indirect Aggression, along with a total score for the combined subscales. Scores range from 34 to 170 with higher scores indicating more aggressive behaviors.
Construct validity was determined through correlated scores with other accepted measures such as Children's Inventory of Anger (r=0.59), and Provocation Inventory (r=0.74) [41].

Problem-solving skills:
The Social Problem-Solving Inventory for Adolescents (SPSI-A) is a 30-item measure with a five-point metric designed to capture adolescent's ability to handle social problems. Several dimensions of problem solving are examined including automatic process, problem orientation and problem-solving skills. The measure was normed with a sample of 1,062 high school students.
A total scale internal consistency score of .95 with subscale scores ranging from 0.78 to 0.95; test-retest coefficients ranging from .68 to .83 over a two-week period suggest good reliability. Evidence of construct validity was demonstrated through high correlation results with other similar measures such as the Problem-Solving Inventory (0.82) [42].

Social skills:
The Social Skills Rating Scales (SSRS) rates several social skills; Communication, Cooperation, Assertion, Responsibility, Empathy, Engagement and Self-Control through a series of 75 items.
Reliability scores range from 0.62 (Assertion) to 0.92 (Self-Control). Convergent construct validity has been established through significant correlations with measures such as the Behavioral Assessment System for Children [43,44].

Statistical analyses:
Each research question was addressed with appropriate statistical procedures. The first question, examining significant change in mental health symptoms, academic self-efficacy, and social skills employed paired t-tests. The second question, examining any differences between treatment and comparison groups was addressed utilizing independent sample t-tests. Lastly, to explore any possible impact of social skills on academic self-efficacy a two-part procedure was followed. Bivariate correlations were used to establish relationships between dependent variables, the control and predictor variables prior to multiple regression analysis. Only significantly correlated variables were used in further analyses. Linear regression was then used to test the impact of social skills on academic self-efficacy.

Results
First, the treatment group produced a number of significant pre to posttest changes ( Table 1). The only measured scale that produced a significant change for the comparison group was the YSR Internalizing Behavior Scale (t= 2.01, m= 5.52 to m=4. 43 Next, independent t-tests were utilized to determine any significant differences between the groups. The only pre-test difference was found on the YSR Aggressive Behavior Scale (t=2.27, p=.026). There were, however, several posttest differences between groups that also demonstrated low-to-moderate effect sizes (Table 2).
Mixed -model repeated measures ANOVA's were conducted in an effort to determine the impact of potentially important demographic factors on any pre-post changes. Due to the number of analyses, the Type 1 error rate was adjusted to .001 with a Bonferroni correction. Results demonstrated that the changes were generally invariant to all measured factors, including age, gender, ethnicity, and guardian's income.
Bivariate correlation analyses between mental health, social skills and academic self-efficacy variables produced several significant results for both the treatment and comparison groups (Tables 3 and 4). Regression analyses were then performed using significant predictors against the dependent academic self-efficacy variable. The comparison group variables failed to produce a significant model. The treatment group however, did produce a significant model as well as several significant predictors (Table 5).

Discussion
This study evaluated an early intervention program delivered in middle schools for at-risk youth. The purpose was to determine if there were changes between the treatment and comparison groups on mental health symptoms, self-regulation skills and academic self-efficacy.
Analyses suggest that the PASS program was effective overall in improving problem-solving, social skills, and academic self-efficacy in at-risk middle school students. The significant change from pre to posttest provides some evidence that the program helps in ameliorating problem behavior. Further, the effect sizes, although not high, indicate that there was a significant difference between the treatment and comparison groups on many of the targeted social skills. These results are consistent with previous program outcome studies suggesting that self-regulation and arts programming improve skills and academic selfconfidence in at-risk youth.
Although there was significant improvement in mental health symptoms, particularly internalizing symptoms, these changes were not generally significant and also occurred within the comparison group. It was surprising that more mental health symptoms were not reduced. Several studies [6,33,37] examining the original program (Prodigy) found significant reductions in mental health symptomology. However, the modification of the program to a school milieu may have impacted this effect or perhaps this school sample presented significantly lower initial mental health issues.
In addition to significant changes supporting overall program efficacy, there were some results suggesting the possible mechanisms of change. The correlations between social skills, internalizing behaviors and academic self-efficacy support the idea that, perhaps, improvement in academic self-efficacy is related to a decrease in some mental health symptoms and, a concomitant increase in certain social skills. While the relationship between academic self-efficacy and a reduction in mental health symptoms has been discussed in previous research, the addition of social skills to the model is an interesting finding [6]. It appears that increased social skills may be an important component to the development of academic beliefs. This concurs with prior studies supporting the importance of social and self-control skills for academic self-efficacy [45,46].
Overall, the findings support positive, significant changes for at-risk middle school students participating in the PASS program as compared to those students simply serving a suspension. While both groups improved in mental health symptoms only the treatment group improved in several important social skills.
A limitation of the study is that it can only describe short-term effects of the program. No longitudinal data were obtained so it is impossible to know if the initial effects of the program continued over time. Another limitation is that youth and families may have been strongly motivated for improvement due to the alternative choice which was school suspension. Future research should include longitudinal designs to measure effects over time and should include a larger sample size to increase external validity.

Conclusions
Despite the noted limitations, the PASS program, modified from the efficacious program-Prodigy implores promise as an alternative to school suspensions for middle school students. The study supports previous research suggesting that the combination of arts and selfregulatory skills results in positive skill and academic self-efficacy changes which can assist at-risk youth. Cawood [1] among others has noted the gap between the knowledge of evidence-supported programs and the use of them by school social workers. In order to be effective  Table 2: Treatment-comparison group post test differences with effect sizes.

SSRS Self-Control YSR Internalizing Behavior
YSR Academic Self-Efficacy .42* -.32* and considered as legitimate change agents, school social workers must adapt and implement effective interventions. However, schools must give school social workers the opportunity to intervene early with atrisk youths and they must be in school to do so.
According to Peckover, Vasquez, Van Housen, Saunders, and Allen [47] the daily roles of school social workers are evolving to include more multilevel tasks (micro, mezzo and macro), including effecting organizational and policy changes. This is an enviable and opportune moment for school social workers to modify unhelpful discipline policies and substitute empirically supported interventions in their stead. Advocacy such as this would have the potential to prevent further problems at-risk youth develop when they are suspended from school.
This study in particular, offers a note-worthy call to adapt other efficacious community-based programs into school-based programs, implement evidence-supported interventions within the schools, and further develop alternatives to the unhelpful policies and noxious consequences of school suspensions and ultimately the Juvenile Justice system.