Variability on the Spectrum: A Self-Monitoring Single-Case Design Study for Students with Autism Spectrum Disorders and Attentional Deficits

This study investigated the differential effect of two treatments, namely self-monitoring of attention and selfmonitoring of performance on spelling accuracy using a multiple-baseline design across participants. Three fifthand sixth-grade male upper-elementary school children diagnosed with Autism Spectrum Disorder in two suburban, public elementary schools in upstate New York were taught a simple spelling strategy. Performance was selfmonitored through the use of a graph, wherein after each session, each student recorded the number of words they spelled correctly. Attention was self-monitored through the use of a MotivAider, a device that vibrated at certain timeintervals. Both visual and regression-based analyses indicated that neither intervention produced a statistically significant treatment effect, with respect to the percentage of words spelled correctly for all participants. However, it seemed that self-monitoring of attention worked slightly better for some of the students. Implications for practice and future research are discussed.


Introduction
Many students with Autism Spectrum Disorder (ASD) have difficulty maintaining focus in academic and related settings due to their inability to self-regulate their learning [1] states that selfregulated learners: 1) set goals, 2) use effective strategies, and 3) monitor their progress. Developing self-regulation is essential for all students, as there are high correlations between self-regulation and improved academic outcomes. For students with ASD, identifying effective interventions and classroom supports in order to develop their self-regulation is paramount to their academic success.
The monitoring of progress "may be one of the most important subprocesses of self-regulation" (Reid, Mason & Asaro-Saddler, 2013). Self-monitoring requires students to reflect back on their behavior in order to assess whether certain targeted goals have been reached [13] When students can properly monitor and regulate their behavior, they are able to function more independently, which is a primary goal for any student with special needs [3] However students with ASD often struggle to monitor and regulate their behavior, particularly in terms of their executive planning and the self-censoring of their behavior [4].
Self-management strategies such as self-monitoring have been shown to be effective in helping students with ASD to develop their self-regulation skills (Carr et al., 2014). In fact, both the National Autism Center and the National Professional Development Center on ASD have designated self-management interventions as one of a limited number of effective, evidence-based practices [5,6]. Selfmonitoring is a relatively easy-to-implement intervention that has been shown to be effective in developing academic self-regulation, as well as other valuable skills [7,8,9].Self-monitoring interventions also can be used across many different types of settings, behaviors, and instructional programs [10] which is important given the range of placement options available to students with ASD. Therefore, selfmonitoring interventions inherently fulfill requirements for interventions for students with ASD to be useful, effective, and generally portable. Since prior research on self-monitoring and ASD has mainly focused on behavioral rather than academic outcomes (e.g. spelling), determining the potentiality for self-monitoring interventions to improve academic outcomes is considered a valuable contribution to the field.
Among educational research, two primary types of self-monitoring have emerged: self-monitoring of performance (SMP) and selfmonitoring of attention (SMA). These two types of self-monitoring, which are distinct in terms of their end goal and time of application, were found to be among the most common types of self-management interventions. When applying SMP, students are tasked with reflecting back upon their academic performance in an effort to increase their on-task attention. Conversely, SMA presupposes that iterative reflection of on-task behavior will improve student performance [11].
Several researchers have explored self-monitoring as a method of remediating learning issues in students with identified disabilities, including Attention Deficit Hyperactivity Disorder (ADHD) and Learning Disabilities specifically in the area of spelling [11][12][13][14][15][16]. A meta-analysis conducted by Graham and Santangelo (2014) found that direct and specific instruction in spelling developed students' spelling (ES=0.54), reading (ES=0.44), and writing abilities (ES=0.94). Therefore, it is a skill worth teaching and remediating.
Although prior research has indicated the effectiveness of selfmonitoring interventions around the academic task of spelling for students with ADHD and LD, investigation into how these may be effective for students with ASD is lacking. The purpose of this study, therefore, was to compare the effects of a self-monitoring of attention intervention and a self-monitoring of performance intervention on the academic performance (i.e., spelling) for students with ASD. This study replicated and extended it by examining the intervention on a different population of students (i.e., students with ASD) and working one-toone with the students in their resource rooms. Specifically, we were interested in these following research questions:

Journal of Mental Disorders and Treatments
Does instruction in self-monitoring of attention (SMA) improve the ability of children with ASD to spell new words?
Does instruction in self-monitoring of performance (SMP) improve the ability of children with ASD to spell new words?
Are there differential effects between SMA and SMP on this population?

Method Setting and Participants
Participants were selected from two suburban public elementary schools in central New York. The percentage of economically disadvantaged students was roughly 14%, while the students with disabilities made up 16% of the total student population for both schools. These schools had demographics of primarily white (~75%) students, with Asian or native Hawaiian/Pacific islander (~14%) making up the largest minority populations. On standardized state exams in ELA, students in these schools performed equally or better than the NYS averages. One of those schools specifically uses a commercialized spelling program.
To select the sample population, special education teachers were asked to recommend students who met the following criteria: (a) documented diagnosis of an Autism Spectrum Disorder, (b) difficulty paying attention, (c) deficits in spelling, (d) ability to write independently with a pencil or pen, and (e) used verbal language as their primary means of communication. Four students were identified for the study; one student, however, reached a ceiling effect during baseline. Therefore, data are reported on three students who completed the intervention.
The study included three Caucasian males. Student 1 was a sixthgrader who had been diagnosed as a high-functioning student with ASD, with specific difficulties in social and emotional functioning. He was educated in a general education classroom with 20 minutes of consultant teacher support provided per day, and received counseling services approximately 30 minutes per week. His full scale IQ scores on the Wechsler Intelligence Scale for Children ® -Fourth Edition (WISC IV) was 98 (M=100, SD=15). As is common in individuals diagnosed with ASD, he was very schedule oriented, and therefore typically had a snack during our sessions due to our sessions occurring during his appointed snack time. This student also was diagnosed with ADHD and an Anxiety disorder. These additional diagnoses were evident during some of the sessions wherein the student would fidget or tap his pen. He displayed a high attention to detail that tended to distract him from central tasks and delay his overall progress. Due to this student's diagnosed anxiety disorder, his motivation and attention were very much a product of his current emotional state. On days when he reported feeling down, sometimes due to academic struggles, he was less attentive and engaged. On days when his emotional state was more positive, he demonstrated more focus and involvement with the spelling tasks. Student 2 was a fifth-grader diagnosed as a high-functioning student with ASD. In addition, this student also had a diagnosis of Tourette Syndrome and Obsessive-Compulsive Disorder (OCD). He was supported throughout the day with a special class for reading and resource room support for math, writing, and social studies/science. He received related services of speech, occupational therapy (OT) and counseling (frequency and duration of services were not provided to the researchers). He scored an 86 in General Conceptual Ability on the Differential Ability Scales ® -II (DAS-II ® ), with a spelling subtest score of 79 (M = 100, SD = 15). His OCD was evident in his need to fix the words he misspelled after the intervention period was over and the researchers were reviewing his work. He demonstrated a strong willingness to participate in the intervention and appeared to enjoy working with the researchers. This student would often converse with the researchers about which words he struggled with and why they were confusing to him. The word lists utilized in this intervention were provided by his classroom teacher and part of his normal spelling and vocabulary instruction. He indicated that he planned on studying the words he got wrong each night. While he displayed higher selfregulation, his severe attentional deficits occasionally slowed his ability to produce spelling words in a timely manner. He also demonstrated multiple letter reversals in his spelling habits. Student 3 was a sixth grader diagnosed with ASD and Pica. This student received speech, occupational therapy, and special education classes in reading and math. Additionally, he also received Relationship Development Intervention (RDI) as part of school-based services. He received a standard score of 66 on the Peabody Picture Vocabulary Test, Fourth Edition (PPVT™-4), scoring in the "extremely low" range. Student 3 has severe difficulties with social and emotional functioning. This was demonstrated when the student would fixate on other tasks and indicate his desire to engage in something else in a loud and somewhat aggressive manner. He also displayed an impulsive and occasionally uncooperative disposition that overall resulted in poor attention skills and poor internalization of any attempted academic strategies. This student had to be reminded about the spelling strategy and general directions before and during each session. This student had cognitive and emotional needs that required more support than the other two students.

Materials
The materials utilized for this study included: a multi-step strategy for spelling new words (see Appendix A); teacher-generated lists of ten spelling words, which changed each week; an academic performance graphing sheet (see Appendix B), in which students recorded the number of words they spelled correctly after each session of SMP; a MotivAider ® ; an attention recording sheet (see Appendix C), in which the student placed a checkmark in the "yes" or "no" box each time they were prompted by the MotivAider ® ; and an attention graphing sheet for the SMA intervention, to record the number of times they were paying attention each session (see Appendix D).
The MotivAider ® was utilized during the six sessions of the SMA intervention. A MotivAider ® is a wearable, lightweight, pager-like device [17]. It can be set to pulse at fixed or variable time schedules. The device shows the timing countdown on its LCD screen and after it vibrates, repeats the sequence again and again until the device is manually shut off [18].

Procedures
Each student worked one-to-one with a doctoral student researcher for each session. The sessions were conducted either during the students' free time or during the time designated for small-group Academic Intervention Services (AIS) Reading. Typically, each student was assessed 2-3 times a week in a separate room or in a private corner of their classroom. Each session lasted approximately 20 minutes; five minutes to get them comfortable and ready, then 15 minutes were spent on the specific spelling-related monitoring task.
Students were provided strategy instruction and baseline data were collected before entering the two intervention phases. The first two days with each student were used to establish rapport and introduce the spelling strategy. After they had successfully internalized the strategy, each student was required to complete four to six baseline sessions, dependent on their chronological position in the multiple baseline design. Two students were randomly assigned to begin with the SMA intervention, while the remaining student was assigned to start with the SMP intervention. Every student received six sessions of each intervention, before switching to the other intervention for an additional six sessions. During the final session, debriefing information was also recorded from the students.

Strategy Instruction
Students were taught a strategy for spelling words correctly based on the list of steps described below. The researcher began by modeling the steps of the strategy on an example word, then prompted the student to use the strategy on the words from their respective lists. They worked together through several words and then the instructor asked the student to try independently, orally working through the steps of the strategy. These steps were mapped out on a chart (see Appendix A) and students were instructed to follow this spelling strategy as they wrote their words during every session. Every spelling task required a list of 10 words that were provided for each student by their respective primary teachers. Lists were updated to include new words based on each students' pacing through their lists. Each student received two days of strategy practice before baseline began.

Baseline
Students were provided with ten spelling words and told to practice spelling their words independently based on the strategy they were taught. Data were collected to determine the baseline levels of attention and academic (spelling) performance.

Self-monitoring of attention (SMA)
The SMA lessons began with the researcher discussing the importance and meaning of paying attention, which was reinforced by the presence of a reminder to pay attention added to the spelling strategy sheet (see Appendix A). Each student was then informed that he would begin to use a technology-assisted procedure to help him pay better attention. Each student was told to ask himself, "Was I paying attention?" immediately upon feeling the vibration of the MotivAider, which was set to vibrate as a prompt for the students. This served as a physical cue to the student to reflect on whether they are attending to on-task behaviors. Each time it vibrated, students had to mark down whether or not they were exhibiting 'on-task behaviors' on an attention recording sheet (See Appendix D). Each student was also prompted to self-record whether he was on task when the vibration occurred by marking a tally sheet in either a "yes" or "no" column. The student then worked through the spelling words with the primary researcher, using the spelling strategy, for fifteen-minute periods. Following each session, each student used an attention graphing sheet to mark the number of times they had been on-task that session (See Appendix D). Each student was observed by the second researcher, who recorded the actual presence or absence of on-task behavior using a time-series interval of 10 seconds (see Appendix E). In each 15 minute session, no student demonstrated more than 2-3 occasions of off-task behavior. There was high reliability (r > .90) between each student's and the observer's recordings of their on-task behavior. Thus, for each session when the SMA intervention was implemented, students demonstrated higher attending skills.
At the beginning of the SMP condition, the researcher discussed the continued importance of practicing spelling words and explained that the student would be using a procedure that may help them spell even better. Additionally, the spelling strategy sheet included a remark reminding students that they will need to record the number of words they had spelled correctly. Students were taught to count the number of times during the fixed 15-minute sessions that their spelling words were written correctly. The researcher first modeled the procedure with a practice list, then asked the child to count independently, and finally graph their total at the end of the session (see Appendix B). The six SMP intervention sessions required each student to use a different academic performance graphing sheet instead, to record the number of words spelled correctly during each session. Across the six SMP sessions, students were able to see their performance and reflect upon their improvement.

Design
A multiple baseline across participants was utilized in this study. The percentage of words spelled correctly is the dependent variable and was observed in a total of 18 sessions for Students 1, 19 for Student 2, and 20 for Student 3. The multiple-baseline across participants design is a strong design [19] as the effectiveness of the treatment can be evaluated at different points in time for different participants. This allowed a functional relationship to be identified between dependent and independent variables more than once, as can be seen in Figure 1; ensuring internal validity and identifying other external effects .
The SMA and SMP interventions, as the independent measures, were counter-balanced in an attempt to account for intervention order effects. The dependent measure used to document changes was academic performance, defined as the percentage of words spelled correctly after each session.

Treatment Fidelity
The two principal researchers independently scored for percentage of correct words; interrater reliability was 100%. To ensure fidelity of implementation, the primary researcher was provided with a script to follow for each session. To check script adherence, the second researcher was also provided with a script to follow for each session. Roughly 94% of the script was adhered to by the primary implementation researcher. The third researcher also attended roughly one-quarter of the sessions, per student, with a copy of the script, checking each step off after completion. She reported that roughly 97% of the steps were followed during her fidelity checks. Generally, deviations from the script were in response to the students, in an effort to maintain a more naturalistic interaction.

Visual analysis
It is recommended to conduct a visual analysis prior to calculating effect sizes for single case experimental data. A visual analysis allows the researcher to help identifying whether there is evidence for a functional relationship between the introduction of a treatment and a change in outcome scores.

Step 1 Baseline Analysis
Evaluation of the baseline phase allows researchers to identify whether the data vary in predictable patterns across participants. Baseline data of Student 3 show an unexpected upward trend. As a result, it is difficult to identify a functional relationship because the percentage of words spelled correctly was improving before either intervention was implemented. The data were examined based on the criterion that 85% of data should be within a 15% range of the average of all data points during that phase [20]. Data do not indicate a high degree of variability, as can be seen in Table 1 and Figure 1.

Mean
Median SD

Step 2 Within-Phase Analysis
It is generally recommended to evaluate within-phase treatment data patterns with respect to the level, trend, and variability of the data. In terms of trend, the data across all participants indicate some variability. In most cases, the data indicate an upward trend and demonstrate a change in the desired direction. However, this can only be observed for Students 1 and 2, as data for Student 3 are more variable.
The mean and median are calculated to examine the level of each phase for all participants.  Shows means, medians, and standard deviations for both SMP and SMA phase scores for each participant. Upon examination, it is clear that the levels for both the SMP and SMA phases are relatively high for Students 1 and 2. Student 3 is the only participant whose performance remained relatively lower compared to the other participants in both conditions.
Overall, most participants' scores fall within one standard deviation of the mean, indicating that variability was generally low between sessions. For the SMP phase, student 1 and student 2 do not the Neuman and Corminck criterian (having 67% and 50% of the scores within the 15% range of the phase mean). For the SMA phase, only students 3 did not meet the criterion (67% of the scores fall outside the 15% range). Data for Students 1 and 2 are more stable during the SMA phase; Student 3 maintained high variability throughout both phases.

Step 3 Between-Phase Comparison
Between-phase comparisons are performed to identify basic effects. Here, a basic effect was identified by comparing data from the baseline phase with the SMP data and then separately with the SMA data. To meet the standards of WWC, the pattern of data in one phase should not be reproduced in another phase . To identify the basic effects, the data were evaluated in terms of 1) the non-overlap between scores in the baseline and each of the intervention phases, 2) the immediacy of effect, as well as 3) changes in level and 4) in slope. Three non-overlap statistics were calculated using the SSDforR package [21], namely the Percentage of all Non-overlapping Data (PND), the Percentage of Data Exceeding the Median (PEM), and Percent of All Non-Overlapping Data (PAND) are reported in Table 3 [22,23,24].  Table 3: PND, PEM, and PAND for Four Participants in SMP and SMA phases.
As can be seen from Table 3, PND for Student 2 equals 0%, suggesting that data in the baseline and SMP phases, as well as in the baseline and SMA phases completely overlap with each other. Therefore, data for Student 2 do not demonstrate a basic effect for either of the two interventions. The PND for Student 1 (17%) and Student 3 (33%) in the SMP phase are low, and therefore indicate a small effectiveness of each intervention. The PND for Student 1 in the SMA phase indicates that this intervention was very effective for him (100%).
Data from the three participants produce a great deal of overlap when measured by PEM between the baseline and SMP phases. In terms of SMA, only Student 1 indicated no overlap between the baseline and SMA phases when measured by PEM (100%). Student 2 had a moderate non overlap between the baseline and SMA phases (83%), and Student 3 had 67% of his data points non-overlapping between phases.
Since both PND and PEM compare data based on one data point, we decided to supplement these indices with PAND because this non overlap index makes all pairwise comparisons [24]. As can be seen from Table 3, the SMA intervention works slightly better for Student 1.
A level comparison of the data across participants indicates a very small change between scores at the end of the baseline and scores at the beginning of the SMP intervention. The largest difference in scores can be observed from Student 3 (Δ = 9.8). In addition, data for all three students indicate a somewhat downward trend in the SMP phase.
A similar pattern can be seen when comparing the baseline phase with the SMA intervention. The largest difference in terms of level was produced by Student 1 (Δ = 13.6). While there was some increase in the percentage of words spelled correct for Students 1 and 2, the differences in level were not high (See Figure 1). In addition, data for Student 3 indicate a downward trend.
These results suggest that data patterns observed do not indicate basic effects of SMP and SMA for all three participants.
Step 4: Do data across phases document experimental control?
This step examines the functional relationship between independent and dependent variables. According to the What Works Clearing House standards, to establish the functional relationship, there should be three demonstrations of basic effects at three different points in time. Based on the visual analysis, it can be concluded that there is no evidence of a functional relationship between the independent and dependent variables. However, student self-monitoring of attention produced slightly better results.
Although no primary evidence for a functional relationship was identified, further analysis of the data was conducted using the regression coefficients to reflect the magnitude of the intervention effects.

Statistical Analysis
Since we were interested in estimating treatment effects of each intervention on the percentage of words spelled correctly, as well as in the differential treatment effect of both interventions, the dummy coding for each set of research questions needed to be different [26]. Therefore, we will first describe the dummy coding for estimating the treatment effect of SMA and the treatment effect of SMP, and then the dummy coding for estimating the differential effects of SMA/SMP on the percentage of words spelled correctly. For more information about dummy coding, graphical display, and interpretation see the article by Moeyaert.

Results
The regression analyses were run using SAS to estimate the treatment effects for SMA and SMP, as well as the differential effects between the interventions. The results are reported by research question.
Does instruction in self-monitoring of attention (SMA) improve the ability of children with ASD to spell new words?
The regression analyses for each of the three participants indicated that SMA was not significantly effective for any of the participants. As can be seen in Table 4  Does instruction in self-monitoring of performance (SMP) improve the ability of children with ASD to spell new words? Similar conclusions can be made for each of the three participants for the SMP intervention. Based on the regression analyses, the SMP intervention did not results in a statistically significant improvement in spelling for any of the participants. For example, the treatment estimates (i.e., 2j ) varied from .36 to 12.35 across three students. Student 2 might have benefitted from SMP slightly more than other students, [β22 = 12.35, t(1) = 1.58, p = .13].
Are there differential effects between SMA and SMP?
An additional set of regression analyses were performed to identify the presence of a differential effect between the SMA and SMP interventions. SMA worked slightly better compared to SMP for Student 1 [t(1) = -1.1, p = .008] For the other participants, there was no statistically significant differential effect of SMP and SMA on words spelled correctly [27,28,29]. As illustrated in Table 5, Student 3 generated more correctly spelled words during the SMP phase (indicated by the negative sign of the estimate). In contrast, the positive nature of the estimates for Students 1 and 2 indicate that they performed better in the SMA condition.

Discussion
Results from both the visual and regression analyses indicate that neither intervention produced a statistically significant treatment effect, with respect to the percentage of words spelled correctly across all participants. While our results are not as statistically powerful as the findings made by, they still indicate a similar trend. As an extension of their foundational study, the methodological and population changes made in the current study resulted in fewer predictable patterns/higher inconsistencies within and between each subject. Further visual analyses indicate a slight increase in words spelled correctly attributable to the strategy instruction alone [30][31][32]. This provides additional support for [27]argument that direct spelling instruction results in beneficial gains in literacy skills. When delivering direct spelling interventions, development of self-regulation through both attention to behavioral and product outcomes may provide similar utility for this population of students.
Because students with ASD, including the students in this study, demonstrate deficits in self-regulation, such interventions worthy of investment. Previous findings involving students solely diagnosed with a learning disability or ADHD have generally indicated a tendency for students to achieve better when monitoring their own performance. However, the SMA intervention in the present study, while not reaching statistical significance, produced slightly better results with respect to spelling production and accuracy for students specifically diagnosed with ASD and comorbid attentional deficits. As the continuous self-monitoring better addressed the absence of behavioral self-regulation in each student's academic work, this population benefitted roughly just as well, if not slightly more, when made to selfmonitor their attention and on-task behavior.

Limitations
Focusing specifically on the self-monitoring behaviors of students with ASD created a unique set of limitations. As autism expert Dr.
Stephen Shore remarked, "If you've met one person with autism, you've met one person with autism. " Primarily, while all of our participants were identified as students with ASD, who met the criteria outlined previously, each student varied greatly in their individual academic abilities. For example, Student 3 had multiple additional diagnoses and was subsequently much lower functioning than the other three participants, making it difficult to analyze his performance in relation to theirs. Additionally, baseline data for Student 1 show an upward trend in words spelled correctly; it is possible the strategy itself influenced spelling gains, thus making the true functional relationship more difficult to isolate. This is supported by the students' scores on standardized intelligence tests. Lastly, there was not a sufficient level of experimental control over the selected word lists, resulting in a ceiling effect during the baseline phase for some participants. Overall, due to our limited sample population, as well as the overarching population from which they were drawn, it is not practical to generalize any specific findings made in this study.

Future Research
Due to the inconclusive nature of the quantitative results drawn from our sample, further study and research are necessary in order to make any informed conclusions with respect to the benefits of selfmonitoring for either or both on-task behaviors and individual performance. While this is a difficult population to study in larger numbers, future interventions on students with ASD and attention deficits should aim to achieve, at minimum, the same sample size as to increase the likelihood of finding statistically significant results from a more stable sample. Also, it would be beneficial to exert more experimenter control in future practice, particularly with respect to the word lists chosen for each participant. This will guarantee that the words are challenging enough to ensure that each intervention has the opportunity to increase each student's current level of achievement and avoid ceiling effects. Since students with ASD have not yet been largely studied across the self-monitoring literature, practitioners should be responsive to the abilities and needs of each individual student when attempting to determine which intervention, if any, may be most beneficial on a case-by-case basis.