From a single-level analysis to a multilevel analysis of single-case experimental designs

https://doi.org/10.1016/j.jsp.2013.11.003Get rights and content

Abstract

Multilevel modeling provides one approach to synthesizing single-case experimental design data. In this study, we present the multilevel model (the two-level and the three-level models) for summarizing single-case results over cases, over studies, or both. In addition to the basic multilevel models, we elaborate on several plausible alternative models. We apply the proposed models to real datasets and investigate to what extent the estimated treatment effect is dependent on the modeling specifications and the underlying assumptions. By considering a range of plausible models and assumptions, researchers can determine the degree to which the effect estimates and conclusions are sensitive to the specific assumptions made. If the same conclusions are reached across a range of plausible assumptions, confidence in the conclusions can be enhanced. We advise researchers not to focus on one model but conduct multiple plausible multilevel analyses and investigate whether the results depend on the modeling options.

Introduction

The use of single-case designs in a variety of different research fields in education as well as the suggested methods to analyze these types of designs have been expanding for decades. In this article, we describe and illustrate one method, namely, the use of multilevel modeling, which provides an appropriate method to analyze and summarize single-case designs (Moeyaert et al., 2013a, Owens and Ferron, 2012, Van den Noortgate and Onghena, 2008). In a single-case study, usually multiple cases, subjects, or participants are involved and repeatedly measured over time (Shadish & Sullivan, 2011). Therefore, in addition to the case-specific estimates, it is useful to develop methods to summarize the results over cases within a particular study. In the first part of this article, we present the basic two-level regression modeling framework that can be used to estimate the treatment effect across cases within studies and the between-case variance of this treatment effect (Van den Noortgate & Onghena, 2003a). We suggest and illustrate a sensitivity analysis approach in which multiple alternative specifications of this basic two-level model are examined. For illustration, we use the dataset of Lambert, Cartledge, Heward, and Lo (2006). In order to allow further examination of external validity and contribute to evidence-based research (Shadish & Rindskopf, 2007), multiple single-case studies measuring the same outcome variable can be combined using the three-level model, which is a straightforward extension of the two-level model. Thus, the second part of this article focuses on the three-level model. We present the basic three-level model assuming no linear trends in which the treatment effect across cases and across studies can be estimated as well as the between-case and between-study variances of this estimate. We will discuss the flexibility of this three-level modeling framework by suggesting multiple alternatives to the basic three-level model. The basic three-level model and alternative specifications of this basic three-level model will be illustrated by summarizing five studies in which a multiple-baseline across participants design was used to investigate the effects of pivotal response training with children with autism.

Section snippets

Two-level model

In single-case experiments, usually more than one case is the focus of interest (Shadish & Sullivan, 2011), such as in the replicated ABAB reversal designs and the multiple-baseline across participants designs. In this first design, there are multiple baseline phases (A phases) and multiple treatment phases (B phases), and the same ABAB design is implemented simultaneously to different participants (see Fig. 1a). In the multiple-baseline across participants design, an AB phase design (with one

Three-level model

The number of published single-case studies is growing rapidly during the last decade, and therefore there is an increasing interest in meta-analyzing these types of studies in order to estimate average treatment effects. The three-level model can be used to synthesize data across cases and across studies. If we pool several studies together, we can examine the generalizability of the results. The synthesis of the studies can inform policy and important decisions can be made based on these

Summary of three-level analysis of single-case experimental data

Table 14 provides a summary of the immediate treatment effect estimates for each proposed model (Models 1 through 4). Notwithstanding each suggested model has its own assumptions, we found similar results for the treatment effect estimates across Model 1 and Model 2 on the one hand and Model 3 and Model 4 on the other hand. The reason is that, in Model 3 and Model 4, a time trend during the treatment phase is modeled, which is not the case in Models 1 and 2. The positive estimated time trend

Discussion

Using the multilevel model (either the two-level or three-level model) to summarize single-case results over cases, over studies, or both has multiple advantages. Multilevel models can provide detailed information regarding the treatment effects (e.g., estimates of case-specific immediate treatment effects, case-specific trend shifts, level shifts across cases and across studies, average trend shifts across cases and across studies, and variance in effects across participants and studies). The

References (44)

  • B.E. Huitema et al.

    Irrelevant autocorrelation in least-squares intervention models

    Psychological Methods

    (1998)
  • R.I. Jennrich et al.

    Unbalanced repeated-measures models with structured covariance matrices

    Biometrics

    (1986)
  • L.K. Koegel et al.

    Setting generalization of question-asking by children with autism

    American Journal on Mental Retardation

    (1998)
  • R.L. Koegel et al.

    Parent education for families of children with autism living in geographically distant areas

    Journal of Positive Behavior Interventions

    (2002)
  • A.D. Koutsoftas et al.

    The effect of tier 2 intervention for phonemic awareness in a response-to-intervention model in low-income preschool classrooms

    Language, Speech, and Hearing Services in Schools

    (2009)
  • J.D. Kromrey et al.

    Determining the efficacy of intervention: The use of effect sizes for data analysis in single-subject research

    Journal of Experimental Education

    (1996)
  • M.C. Lambert et al.

    Effects of response cards on disruptive behavior and academic responding during math lessons by fourth-grade urban students

    Journal of Positive Behavior Interventions

    (2006)
  • K.E. Laski et al.

    Training parents to use the natural language paradigm to increase their autistic children's speech

    Journal of Applied Behavior Analysis

    (1988)
  • S.D. McKnight et al.

    A double bootstrap method to analyze linear models with autoregressive error terms

    Psychological Methods

    (2000)
  • M. Moeyaert et al.

    Three-level analysis of single-case experimental data: Empirical validation

    The Journal of Experimental Education

    (2013)
  • M. Moeyaert et al.

    The influence of the design matrix on treatment effect estimates in the quantitative analyses of single-case experimental design research

    (2013)
  • M. Moeyaert et al.

    Three-level analysis of standardized single-case experimental data: Empirical validation

    Multivariate Behavior Research

    (2013)
  • Cited by (0)

    This research is funded by the Institute of Education Sciences, U.S. Department of Education, Grant number R305D110024 and Research Foundation - Flanders. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education or the Research Foundation - Flanders.

    View full text