The impact of stimulus preference, order-effects, and treatment component omission in evaluating treatment integrity

Abstract

Prior research on treatment integrity has focused either on the lack of measurement of the independent variable or on methods to increase overall levels of treatment integrity. Little research has focused on the effectiveness of common interventions when implemented with less than perfect integrity. The current investigation evaluated the effectiveness of using differential reinforcement of alternative behavior (DRA) and prompting to increase math completion for 36 early elementary students. Treatment was evaluated when both components were implemented, when only reinforcement was implemented, when only prompting was implemented, and when neither was implemented. In addition, preferences for either attention or escape and order-effects of conditions were evaluated. Results indicated treatment was effective at all levels of implementation compared to baseline. However, when preferences for escape and attention were evaluated, analysis revealed individuals who preferred escape responded best when both treatment components were implemented, whereas for individuals who preferred attention, all treatment conditions were equally effective. In addition, results evaluating order effects indicated that exposure to either prompting or reinforcement prior to baseline significantly increased math completion as well as exposure to reinforcement in the first condition.

Introduction

There is an extensive research literature demonstrating the efficacy of behavioral interventions to address challenging behavior and learning across a diverse range of populations, target concerns, and contexts (Heyvaert et al., 2014; Wilson and Lipsey, 2007). Although the research in this area is compelling in demonstrating functional relationships and treatment gains, many of the studies have been conducted in tightly controlled settings (e.g., hospitals, in-patient units, and specialized schools; Kurtz et al., 2003; Kurtz et al., 2013) or have been implemented in more natural environments by research staff specifically tasked with treatment implementation (Bloom et al., 2013; Greer et al., 2013). Studies employing a dedicated research staff and/or a controlled setting are ideal for understanding if an intervention is effective, but are not ideal for determining the extent to which treatment efficacy will generalize to natural environments (e.g., schools) where treatment is less likely to be implemented with such high integrity. Uncontrolled competing stimuli (e.g., classmates) and the necessity for treatment to be implemented by individuals with less rigorous training (e.g., teachers and parents) are common barriers to treatment implementation in naturalistic settings. These environmental challenges often lead to decrements in treatment integrity (i.e., the degree to which a treatment is implemented as intended; Noell et al., 2005; Peterson et al., 1982).

Understanding the effects of decreased treatment integrity is important because it is unlikely that interventions will be implemented in the natural environment with sustained perfect integrity. The complexity of the natural environment may lead to lower treatment integrity. In addition, behavioral interventions often include multiple components, such as extinction and reinforcement (Petscher et al., 2009), making the interventions complex. The complexity of both the natural environment and the treatments themselves may lead to treatment integrity failures (Gresham, 1989). Therefore, it is important to study and understand the effects of implementation errors on intervention efficacy and how these errors affect behavioral interventions.

Research that systematically manipulates treatment integrity to understand how effective treatments are when components of the intervention are completely omitted or implemented with less than perfect integrity has been limited. At a macroscopic level, it is clear that treatment implementation is important, but little data are available demonstrating how important it is or to what degree errors in integrity effect treatment outcomes. While most research conducted in school settings has demonstrated that higher levels of integrity of treatment implementation are associated with greater treatment gains (Noell et al., 2005), other studies suggest that some treatments are robust to reduced treatment integrity (Gansle and McMahon, 1997). Research in this domain has been limited, with only the few studies building systematically on prior treatment integrity research. Some of this research has focused on integrity as it relates to the treatment component of extinction. Extinction is the termination of reinforcement following a behavior and is a common treatment component included in behavioral interventions implemented in school settings (Janney et al., 2012; Stahr et al., 2006). Despite the greater number of studies examining the effects of integrity errors with extinction, the results have still not been strongly convergent in their findings (Athens and Vollmer, 2010; Mazaleski et al., 1993; Shirley et al., 1997). Some studies have demonstrated that extinction was necessary to produce positive outcomes when using differential reinforcement to decrease problem behavior (Mazaleski et al., 1993); whereas other studies have found extinction to be inconsistently essential to successful treatment (Athens and Vollmer, 2010; Shirley et al., 1997). The studies vary enough in their procedural details that drawing conclusions regarding the necessity of extinction or the level of implementation needed has proven difficult.

Research has also examined the impact of treatment integrity errors in treatments that have included differential reinforcement of alternative behavior (DRA). Differential reinforcement of alternative behavior is when reinforcement is discontinued for the problem behavior (extinction) but reinforcement is provided contingent on an appropriate alternative behavior (e.g., functional communication response, compliance; Vollmer et al., 1999). DRA is a common treatment component in school-based interventions (Petscher et al., 2009). In a study by Northup et al. (1997), participants were initially exposed to DRA with complete treatment integrity. Subsequently, participants were exposed to conditions in which either reinforcement, punishment, or both were delivered with 50% or 25% integrity. Treatment at 50% implementation remained effective and only a slight increase in problem behavior emerged at 25% integrity. Subsequent research extended this study by introducing commission errors and omission errors (Vollmer et al., 1999). Commission errors consisted of delivering reinforcement for problem behavior (i.e., integrity errors for extinction) whereas omission errors consisted of failing to deliver reinforcement for appropriate behavior (i.e., integrity errors for DRA). All participants were initially exposed to treatment at complete integrity, which was effective. Subsequently, the intervention remained effective when 50% of the intervals involved either a commission or omission error. While the available research suggests that DRA remains effective when integrity falls as low as 50% implementation of the planned scheduled, the available studies are confounded by order effects. Participants were all exposed to complete treatment integrity long enough for stimulus control to be achieved before exposure to lower levels of treatment integrity. Stimulus control emerges when an individual's behavior has a history of being reinforced when they emit a specific behavior in the presence of specific stimuli. If after this learning history the individual is more likely to emit that behavior in the presence of those stimuli, stimulus control has been established. From the existing research it is not possible to tell whether DRA at lower levels of integrity would be effective without this prior learning history.

Subsequent research has also demonstrated that sequence effects are evident in the study of treatment integrity (St. Peter-Pipkin et al., 2010). St. Peter-Pipkin and colleagues demonstrated that omission errors, failures to reinforce desired behavior, were less damaging to treatment efficacy than commission errors, reinforcing problematic behavior. Treatment remained effective when reinforcement was delivered for only 60% of planned occasions. However, when treatment integrity fell below 40%, participants engaged in lower levels of appropriate responding. When commission error occurred, treatment was less effective. St. Peter-Pipkin and colleagues also evaluated whether treatment integrity errors had different effects when conducted following a condition with either baseline contingencies or a phase with 100% treatment integrity. Results suggested treatment following a phase of 100% integrity was less likely to break down when exposed to treatment integrity errors than treatment that followed a baseline phase. The protective effect of exposure to high treatment integrity may be a result of developing a strong history of reinforcement for appropriate behavior; making the appropriate behavior more resistant to extinction when treatment integrity errors occur. It is important to note that this study suffers from the same limitation as earlier research in that all participants were exposed to complete treatment integrity before exposure to conditions with lesser levels of implementation. As a result, the importance of this prior learning history cannot be determined.

Despite the growing use of function based interventions in schools (Beavers et al., 2013; Individualized with Disabilities Act [IDEA], 1997), function (i.e., reinforcer maintaining the problem behavior) has not been evaluated as a potential moderator of treatment integrity. It is possible that treatment integrity failures would differentially affect treatment outcomes depending on the function of behavior. For example, common classroom interventions often include a combination of reinforcement (e.g., praise provided contingent on compliance) and prompting procedures (e.g., least-to-most prompting provided contingent on noncompliance). If escape from work demands is preferred by the student, failures in the prompting component might be more detrimental than failures in praise and accuracy feedback. On the other hand, if the student's behaviors are commonly maintained by attention (e.g., attention is the reinforcer maintaining the behavior), failures to deliver reinforcement (e.g., praise) may be more a detrimental integrity failure. For the individual that prefers escape, prompting allows the teacher to implement a version of escape extinction. On the other hand, for the individual that prefers attention, the functional reinforcer of praise may be the critical component of intervention. Although this paradigm makes sense from a theoretical perspective, it has not been evaluated empirically.

The current study sought to examine three research questions. First, what is the effect of treatment integrity errors in prompting and reinforcement in a DRA intervention aimed to increase compliance in an analog instructional setting? Second, do preferences for either escape or attention play a role in differentially affecting correct responding when there are treatment integrity failures in prompting and reinforcement delivery? Third, how does prior exposure to one condition affect responding in subsequent conditions (order effects)? A within-subject repeated measures design where each participant was exposed to each condition was used to evaluate the impact of treatment integrity errors in prompting and reinforcement. A between-subject design was used to answer the two remaining research questions.