Elsevier

Early Childhood Research Quarterly

Volume 48, 3rd Quarter 2019, Pages 186-197
Early Childhood Research Quarterly

Evaluating the viability of a structured observational approach to assessing early self-regulation

https://doi.org/10.1016/j.ecresq.2019.03.003Get rights and content

Highlights

  • We examined viability of an observational assessment of self-regulation (PRSIST).

  • PRSIST ratings showed good associations with task-based and adult-report measures.

  • PRSIST and task-based scores, but not adult report, captured age-related change.

  • PRSIST and task-based scores, but not adult report, related to school readiness.

  • PRSIST is unique in its high ecological validity and direct relevance to practice.

Abstract

There is now clear evidence that young children’s abilities to self-regulate set in motion developmental trajectories that are longitudinally influential, that self-regulation is malleable in childhood with flow-on implications into adulthood, and that efforts to improve self-regulation are enhanced when tailored and adaptive to children’s developmental progress. Yet there are significant concerns regarding current approaches to assessing self-regulation, potentially limiting our capacity to respond to children’s needs and support their development. In response, this study examined the viability of a novel approach to assessing young children’s cognitive, behavioral, and social-emotional self-regulation that uses observer ratings in semi-structured activities. Eighty 3- to 5-year-old children were observed in their preschool in a small group task (a memory card game with other children) and in two individual tasks (an outdoor obstacle course and a curiosity box guessing game). Exploratory factor analysis of resultant Preschool Situational Self-Regulation Toolkit (PRSIST) assessment ratings yielded a two-factor solution which corresponded to cognitive and behavioral self-regulation. Comparison of these observation ratings with task-based (Head–Toes–Knees–Shoulders; HTKS) and adult-report measures of self-regulation (parent and teacher report on the Child Behavior Rating Scale; CBRS) showed strong association with HTKS scores and more modest relations with CBRS ratings. Unlike adult report, however, PRSIST and HTKS were also developmentally sensitive, both showed robust associations with children’s school readiness, and neither revealed systematic differences across genders. These findings offer encouraging support for the PRSIST approach, which has high ecological validity and direct relevance to those working with children.

Introduction

By the end of the preschool years a well self-regulated child can sustain their attention and resist distraction, endure temptations and delay gratification, wait their turn, consider the consequences of their actions, and persist with challenging activities. They are often able to initiate or cease behaviors that conflict with their immediate preferences or impulses, such as listening to the views of other children in group time despite a desire to focus on their own interests. Self-regulation can broadly be considered as the ability to control our thinking, behavior, social interactions, and emotional reactions, despite contrary urges and impulses (occasionally referred to as self-control; Moffitt et al., 2011). While the segmentation of self-regulation (i.e., as some permutation of cognitive, behavioral, and social-emotional dimensions) and its delineation from related abilities remains debated (e.g., Rademacher & Koglin, 2018), one prominent proposal suggests that successful self-regulation in complex real-world contexts involves: goal setting, to behave in a self-regulated way; motivation, to remain self-regulated when this becomes difficult; and capacity to sustain attention toward goals, while resisting contrary impulses and distractions (with this cognitive control enabled by executive functioning; Hofmann, Schmeichel, & Baddeley, 2012).

A myriad of studies underscore the foundational role of self-regulation for influencing developmental trajectories related to: school readiness and academic success (e.g., Ponitz, McClelland, Matthews, & Morrison, 2009); risky lifestyle choices in adolescence (such as substance use, self-harm; Howard & Williams, 2018); and health, wealth, substance abuse, and criminal offending in adulthood (Moffitt et al., 2011). Good self-regulation thus appears to enable important kinds of freedom; freedom from needing constant direction from others, from maladaptive and inappropriate impulses, and from unnecessary distraction. There is also mounting evidence that early self-regulation, in particular, is important for establishing initial developmental trajectories that are widely and longitudinally impactful (Howard and Williams, 2018, Moffitt et al., 2011). Rather than these trajectories being fixed early, however, there is compelling evidence for the malleability of self-regulation in childhood, with flow-on effects for diverse outcomes in adulthood (Moffitt et al., 2011). Indeed, it has been suggested that early intervention may produce more pronounced, stable, and lasting positive change across the lifespan (e.g., Wass, Scerif, & Johnson, 2012), and provide better return on investment (Heckman, 2006), than comparable efforts later in life.

Uncertainty remains, however, as to how such changes are best effected and supported. In relation to the cognitive control mechanisms underlying self-regulation (i.e., executive functions; Hofmann et al., 2012), current findings indicate that children benefit more from adaptive training tailored to their current and changing ability, than from less-differentiated and non-adaptive intervention approaches (Chacko et al., 2014; Dunning, Holmes, & Gathercole, 2013; Green et al., 2012). For instance, Dunning et al. (2013) found that increasing the demand on working memory in line with children’s expanding capacities led to greater improvements than training with a static working memory demand. Indeed, this idea permeates contemporary educational curricula and theory, and has a strong evidence base across diverse populations and domains (Kremer, Brannen, & Glennerster, 2013). In light of these findings, efforts to foster self-regulation may be improved by a clear understanding of individuals’ developmental progress and ongoing change, in ecologically valid situations, and across the range of domains to which self-regulation might apply (i.e., cognitive, behavioral, socio-emotional).

There are two predominant approaches for appraising self-regulation in childhood: adult report and task-based. Adult report relies on adults who are familiar with the child – typically a parent or educator – and asks them to rate the frequency, severity, or typicality of a child’s everyday self-regulatory behavior (e.g., persists in difficult tasks, concentrates when working on a task; Bronson, Goodson, Layzer, & Love, 1990; Howard & Melhuish, 2017). This approach has the benefit of targeting real-world self-regulatory behaviors that are ecologically valid and predict later-life outcomes (Sylva, Melhuish, Sammons, Siraj-Blatchford, & Taggart, 2004). Adult report also has the benefit of generating extensive data at minimal time and cost. Yet the extent to which these ratings accurately reflect children’s current self-regulatory abilities remains unclear. For instance, despite the fact that self-regulation develops rapidly across the preschool years – as is clear when comparing the behavior of a child at 3- and 5-years of age – adult-report ratings often portray little developmental change (e.g., Howard & Melhuish, 2017). This discrepancy may be attributable to the subjective nature of adult-reported ratings, such that respondents likely reference a child to their same age-peers (e.g., ‘this child persists well in difficult tasks for a 3-year old’). As a consequence, the data generated by this approach are more reflective of a child’s self-regulatory ranking within their context and age group, which is more stable but is less useful for supporting change or informing interventions than an objective index of development.

Furthermore, the typically modest associations between parent- and educator-reports of children’s self-regulatory capacities – despite being on the same child and using the same scale – across a range of child development dimensions further highlights the subjectivity or situational inconsistency inherent in this approach (Goodman, 2001; Murray, Ruble, Willis, & Molloy, 2009). Indeed, these inherent methodological problems may explain findings that adult-report measures show different patterns of association with other child outcomes – and uniquely indicate gender differences, such that girls are usually rated as better self-regulators than boys (Matthews, Ponitz, & Morrison, 2009) – than more direct self-regulation measures (Schmitt, Pratt, & McClelland, 2014).

An alternative tradition in assessing early self-regulation is direct task-based approaches, in which children are asked to undertake an activity that is believed to require self-regulation to successfully complete. Amongst the most highly researched of these are the Marshmallow and Head-Toes-Knees-Shoulders (HTKS) tasks. In the Marshmallow task (Mischel, Shoda, & Rodriguez, 1989), children are presented with a marshmallow and are informed that they will receive a second one if they can refrain from eating the initial marshmallow for a period of time. In HTKS (Ponitz et al., 2009), a child is taught a correspondence between two or more body parts (e.g., head-knees) and then must touch the body part that is opposite to what they are directed to touch (e.g., touch their head if told to touch your knees). In both cases, these tasks require children to overcome natural impulses (e.g., to eat a sweet or to carry out an instruction as given) and instead behave in accordance with the rules of the situation.

Strengths of task-based approaches lie in their structured situation and direct assessment of children’s abilities, which contribute to their strong psychometric properties (McClelland et al., 2014), sensitivity to age- and intervention-related changes (Tominey & McClelland, 2011), and association with academic and non-academic outcomes (e.g., Ponitz et al., 2009; Schlam, Wilson, Shoda, Mischel, & Ayduk, 2013). Yet task-based self-regulation measures diverge in the extent to which they approximate the complex and often affective real-world situations in which children typically must self-regulate. While the Marshmallow Task does require children to resist a tangible and affective impulse that is routinely experienced in life (i.e., to wait before eating), HTKS assesses children’s ability to follow instructions in a novel task without this affective component. As such, the extent to which performance across these tasks genuinely index children’s everyday self-regulatory abilities (e.g., ability to share, take turns, resist emotional outbursts) may vary.

Indeed, rather than a measure of self-regulation, it may be that tasks like HTKS measure executive function through their requirements, for example, to maintain rules in mind (i.e., in working memory) and perform opposite actions (i.e., via inhibition). This construal of HTKS (and other similar paradigms) as indexing cognitive control aspects of self-regulation, and not self-regulation per se (which, at a minimum, additionally involves goal-setting, motivation, and problem solving; Hofmann et al., 2012), is supported by findings that executive function tasks show similar patterns of results to task-based ‘self-regulation’ measures (Fitzpatrick, McKinnon, Blair, & Willoughby, 2014; Miller, Nevado-Montenegro, & Hinshaw, 2012).

An alternative and emerging approach to self-regulation assessment in childhood uses observation in ecologically valid situations to combine some of the strengths of task-based (i.e., greater objectivity, developmental sensitivity) and adult-report approaches (i.e., a focus on authentic self-regulatory behaviors, capturing distinct dimensions and applications of self-regulation), while minimizing their limitations. The importance of ecological validity – that is, whether measures, methods, and materials approximate the real-world conditions of the factor(s) under investigation – is illustrated by research suggesting that task-based executive function measures account for only one-fifth of the variance in everyday executive ability (Chaytor, Schmitter-Edgecombe, & Burr, 2006). While measures such as HTKS similarly strive to approximate and leverage familiar situations (e.g., ‘Simon Says’), their individual, highly structured, and less-affective nature means that they miss much of the complexity, interactivity, and emotionality that is prevalent in the daily situations in which children are expected to self-regulate.

By contrast, observational tools seek to emphasize both psychometric and ecological validity in generating profiles of self-regulation across its various areas of application (e.g., cognitive, behavioral, social-emotional). For example, the Response to Challenge (RCS) scale evaluates self-regulation as a child completes routine physical activities in an obstacle course (Lakes, 2012). Observers rate a child’s attentiveness, assertiveness, and perseverance. In another paradigm, the Preschool Self-Regulation Rating Assessment (PSRA) engages children in a set of executive-function-like tasks (e.g., do not peek as a present is wrapped) but rates their attention, impulse control, and regulation of arousal, instead of accuracy or response times typical of executive function task indices (Smith-Donald, Raver, Hayes, & Richardson, 2007). The Observed Child Engagement Scale (OCES; Rimm-Kaufman, Curby, Grimm, Nathanson, & Brock, 2009) observes similar dimensions within play or learning situations (e.g., engagement, compliance).

While these observation-based measures are responding to important issues of validity and developmental sensitivity, they are not without issue. For example, to generate a reliable self-regulatory index on the RCS, multiple live observers are required for each child (Lakes, 2012). In the PSRA, use of executive-function-like activities raises similar questions as some task-based approaches regarding children’s emotional and motivational investment in these non-typical situations. For the OCES, when unstructured free play is observed, children’s self-selection into different types, durations, and complexities of free play makes it difficult to compare ratings across children; this may explain why OCES ratings did not relate to any child outcomes investigated in a comparison of adult-report, task-based, and observational approaches (Schmitt et al., 2014). Notwithstanding these challenges, the potential benefits driving these approaches are worth pursuing. If developmentally sensitive and valid indices of children’s early and everyday self-regulation can be captured through direct and structured observation of their real-world activities, our capacity to respond to children’s needs and support their development would be enhanced.

To explore the viability of a structured observational assessment of children’s early self-regulation development, and in response to issues identified above, the Preschool Situational Self-Regulation Toolkit (PRSIST) assessment was created. First, to ensure consistency across the situations in which children would be observed, semi-structured activities were selected on the basis of offering sufficiently high self-regulatory challenge (e.g., children needing to persist in a challenging task, overcome frustration, stay within the rules of the activity) to evoke stable individual differences in children’s self-regulatory development. Further, to preserve ecological validity, the activities also had to be compatible with early childhood contexts and routines, reflect the diversity of experiences children have in these settings (i.e., group, individual), and be similar to common activities that young children already engage in. Based on these considerations, the first activity selected was a group-based memory card game. It was anticipated that this activity would introduce a high degree of challenge across the cognitive (e.g., sustain attention, resist distraction), behavioral (e.g., take turns, follow the rules), and social-emotional domains (e.g., cope with frustration) within which self-regulation applies, as a consequence of its semi-structured and group format, opportunities for impulsive behaviors, and need for sustained attention.

Whilst the group-based activity has obvious advantages – such as its approximation of the dynamic, complex, and emotionally laden contexts in which children must routinely self-regulate – it stands in contrast with the overwhelming precedent in the literature to examine children individually. Further, given the possibility that self-regulatory capacity might differ across group and individual contexts, it was also decided to select an individual activity. An obstacle course activity was included on the basis of its anticipated self-regulatory challenge (e.g., persist if initially unsuccessful, follow instructions and sequences). However, given that difficulties emerged in obtaining reliable self-regulation ratings from children’s obstacle course performance using the RCS (which may be influenced by conflation of physical and self-regulation items in the RCS, rather than the activity per se; Lakes, 2012, Lakes and Hoyt, 2004), a briefer curiosity box guessing game was also included. It was intended that one individual activity would be selected from these two options, if necessary, to supplement the group activity.

An associated scale was then created based on existing literature and the observation of children during the activities, to capture dimensions of cognitive (e.g., attention, distraction), behavioral (e.g., self-directedness, behavioral control), and social-emotional self-regulation (e.g., emotional reactions and recovery, helping). This mirrors the approach of adult-report measures, and contrasts with the single-index result of other task-based measures, in that it provides a profile of development across a range of self-regulatory dimensions. Yet to ensure comparable sensitivity to age- and intervention-related change as task-based approaches, an iterative process of developing and refining descriptors and exemplars of each behavior was also conducted so that self-regulatory behaviors could be clearly and consistently situated on a developmental continuum. Specifically, after initial item creation, multiple rounds of rating occurred with seven raters viewing video recordings of children participating in the activities, and then independently rating using a working draft of the scale. Subsequent discussions of ratings, interpretations, and rationales contributed to refining the PRSIST items, descriptions, and developmental continua embedded in the scale. Finally, to ensure maximum objectivity and reliability across raters, a structured process for training and evaluation was created. This potentiates future use of the tool not only by researchers, but also by those who spend the majority of time with young children (e.g., educators). In this respect, the PRSIST assessment contrasts sharply with the vast majority of existing assessments, which are explicitly reserved for trained researchers (e.g., PSRA; Smith-Donald et al., 2007) or yield results that have less-clear application to educational practice (i.e., do not provide explicit guidance on the specific areas requiring support, which are needed to act upon the data; Ponitz et al., 2009).

The current study thus evaluated the viability of a structured observational approach to self-regulation assessment. Specifically, we aimed to: (1) determine whether an observation approach could yield insights about discrete yet correlated cognitive, behavioral, and social-emotional dimensions of self-regulation; (2) evaluate psychometric properties of the PRSIST assessment; (3) examine the level of agreement between observer ratings of self-regulation with task-based and adult-report assessments; and (4) examine age trends across each of the measurement approaches. It was expected that the PRSIST ratings would yield three related yet distinct self-regulation factors with good psychometric properties. It was also expected that, despite their clear differences, the PRSIST assessment would more strongly relate to an existing task-based measure (i.e., HTKS) than an educator- or parent-report measure. This was expected due to: our additional expectation that the PRSIST assessment would detect age-related changes in self-regulatory development (like task-based measures, but unlike adult-report measures); and that PRSIST was seen as an intermediary between task-based (i.e., increasing the objectivity of adult reports) and observational approaches (i.e., using a questionnaire-like scale of children’s everyday behaviors to guide ratings). In addition to these expectations, we also: included an index of children’s school readiness as an external validation of the various self-regulatory measures in the study; and explored possible gender differences in children’s self-regulation, which have been found more commonly in adult-report measures (Matthews et al., 2009).

Section snippets

Participants

Participants were 80 3- to 5-year-old children (Mage = 4.46, SD = 0.73; range = 3.10 - 5.88) who were enrolled at one of seven preschool services in or proximal to a metropolitan city in Australia. More than 80% of 3- to 5-year old children in Australia are enrolled in pre-school education (which is in line with the OECD average; OECD, 2018). Preschools ranged in their statutory government quality ratings against the National Quality Standard (ranging from Meeting to Exceeding this standard).

Initial exploratory data analysis

Missing data were explored in terms of the extent and patterns of missingness. Four of the children did not have complete PRSIST data due to early departure from their preschool on the day of assessment. Inspection of these cases against available HTKS data showed that while these children were older than the mean age of the sample, their scores were within the overall distribution, each of the values fell within the 95% CI of the sample mean, and their scores were not clustered at an extreme

Discussion

The current study provides strong initial evidence of the viability – indeed, strength – of a structured observational approach to self-regulation assessment. This is in contrast to the challenges and limitations of previous attempts to adopt this approach (e.g., Lakes, 2012). In particular, the current study demonstrated the PRSIST assessment could clearly identify two distinct yet related cognitive and behavioral self-regulation dimensions using an observation approach. In addition, the

Conclusion

While these areas of further investigation are indeed important, the direction and strength of the current results are clear. Specifically, the current study evidenced the viability of a structured observational approach to self-regulation assessment, which displayed very strong concurrent and predictive validity, and good to excellent psychometric reliability. Viability of this approach also proffers wide-ranging possibilities unique to this approach, such as: the ability to put the tool in

Funding source

This work was supported by philanthropic financial support from the Colman Foundation. This funding body had no role in the design, conduct, or reporting of this research.

Declarations of interest

None.

References (45)

  • ABS (Australian Bureau of Statistics)

    Information paper: An introduction to Socio-Economic Indexes for Areas (SEIFA) (cat. no. 2039.0)

    (2008)
  • M.A. Adams et al.

    An adaptive physical activity intervention for overweight adults: A randomized controlled trial

    PLoS One

    (2013)
  • V.R. Alferes et al.

    SPSS programs for the measurement of nonindependence in standard dyadic designs

    Behavior Research Methods

    (2009)
  • M.R. Basso et al.

    Practice effects on commonly used measures of executive function across twelve months

    The Clinical Neuropsychologist

    (1999)
  • B.A. Bracken

    Bracken School Readiness Assessment [measurement instrument]

    (2007)
  • M.B. Bronson et al.

    Child behavior rating scale

    (1990)
  • A. Chacko et al.

    A randomized clinical trial of Cogmed Working Memory Training in school-age children with ADHD: A replication in a diverse sample using a control condition

    Journal of Child Psychology and Psychiatry

    (2014)
  • T.L. Davis

    Gender differences in masking negative emotions: Ability or motivation?

    Developmental Psychology

    (1995)
  • D.J. Dunning et al.

    Does working memory training lead to generalized improvements in children with low working memory? A randomized controlled trial

    Developmental Science

    (2013)
  • E. Fink et al.

    Validation of the Peer Social Maturity Scale for assessing children’s social skills

    Infant and Child Development

    (2013)
  • E. Fink et al.

    Young children’s affective responses to another’s distress: Dynamic and physiological features

    PLoS One

    (2015)
  • J. Heckman

    Skill formation and the economics of investing in disadvantaged children

    Science

    (2006)
  • Cited by (26)

    • Self-regulation in toddlers and the emergence of pre-academic disparities

      2022, Infant Behavior and Development
      Citation Excerpt :

      Notably, these two latent variables were significantly and positively correlated with one another, as were many of the individual indicators across factors at the bivariate level in Table 1. These positive associations between researcher- and parent-reported measures of self-regulation are consistent with past research with two- to three-year-old children (Geeraerts et al., 2019) as well as among preschoolers and kindergarteners (Howard et al., 2019; Ponitz et al., 2009). Table 2.

    • Dynamic assessment of self-regulated learning in preschool

      2022, Heliyon
      Citation Excerpt :

      Therefore, when choosing a measure to assess SRL, the instrument's potential and the learners' developmental specificities should be taken into account to ensure that a deeper understanding of the overt and covert processes is gathered. Other commonly used instruments to assess SRL in young children are the observation measures (Dörrenbächer and Perels, 2018; Howard et al., 2019; Nader-Grosbois et al., 2008; Whitebread et al., 2009). These present some advantages because they do not rely on children's self-report, they provide objective information as a result of external assessment and they have applicability to preschool practices.

    • Enriching parent-child play through supported play groups to foster children's self-regulation: Protocol for a randomised control trial

      2022, International Journal of Educational Research
      Citation Excerpt :

      The Preschool Situational Self-Regulation Toolkit (PRSIST; Howard et al., 2020) assessment is an observational measure of self-regulation. Although the complete assessment consists of two activities, only the group Memory Card Game will be adopted for this study, which has good internal consistency for both cognitive self-regulation (α = 0.88) and behavioural self-regulation (α = 0.81, Howard, Neilsen-Hewett, de Rosnay, Vasseleu & Melhuish, 2019). In this game, a group of 3–5 children are invited to play a memory game, wherein they try to find matching pairs amongst an array of 16 cards placed face down.

    • Associations between South African preschoolers’ routine physical activity, self-regulation and psychosocial well-being

      2021, Mental Health and Physical Activity
      Citation Excerpt :

      This highlights one of the limitations with the use of subjective ratings of self-regulation and psychosocial well-being. However, previous studies have shown that teacher-report measures are typically more predictive and highly correlated with objective measures of self-regulation compared to parent-report measures, emphasising the a strength in the current study (Howard, Neilsen-Hewett, et al., 2019). Nevertheless, future studies should aim to include both objective and subjective measures of self-regulation to replicate these results.

    View all citing articles on Scopus
    View full text