Reliability and plasticity of response inhibition and interference control
Highlights
► We examine test-retest-reliability, internal consistency and plasticity of tasks measuring inhibition-related functions. ► Most measures yield good test-retest-reliability and internal consistencies. ► No systematic within- and between-session changes are present. ► Therefore these measures are able to represent inhibition-related traits. ► Results are relevant for pathophysiological, genetic and pharmacological research.
Introduction
A hallmark feature of cognitive control is the ability to inhibit automatic or inappropriate reactions and to resolve response conflict stemming from irrelevant or distracting stimuli or stimulus features (Barkley, 1997, Miller and Cohen, 2001). While the term inhibition has different meanings in different contexts of psychology and neuroscience (Aron et al., 2007), inhibition is often defined as the ability to cancel a prepotent response or to suppress an action which is inappropriate, irrelevant or no longer required (Friedman and Miyake, 2004, Nigg, 2000, Schachar et al., 2000). Different paradigms have been used to measure different forms of inhibition, including motor response inhibition, reflex inhibition and interference inhibition.
Prominent experimental paradigms of motor response inhibition include the stop-signal (Logan, Cowan, & Davis, 1984) and the go/no-go task (Meere, Stemerdink, & Gunning, 1995). In the go/no-go task a response has to be either executed or not depending on the go or no-go-signal and the task thus measures selective motor response inhibition. In the stop-signal task a motor response has to be withheld when a stop-signal appears unpredictably a short time after the go-signal. The stop task thus measures a later stage of inhibition, where the go response is already triggered by the go-signal and on its way of execution, but needs to be retracted (Rubia et al., 2001). A different variant is the antisaccade task that measures the inhibition of a prepotent, automatic saccadic response (Hallett, 1978). A more cognitive form of inhibition is interference inhibition which measures the ability to resolve response conflict due to irrelevant but incompatible and therefore interfering stimulus features that, if not inhibited, will lead to an erroneous response, such as the Simon (Simon & Rudell, 1967), Eriksen flanker (Eriksen & Eriksen, 1974), and Stroop (Stroop, 1935) tasks. The Continuous Performance Test (CPT) is more typically considered a measure of sustained attention, where targets have to be detected among non-targets, but also contains an inhibitory component, viz. the suppression of a response to irrelevant targets (Cornblatt et al., 1988, Rosvold et al., 1956, Soreni et al., 2009).
While these tasks differ with respect to their specific cognitive demands and may be grouped according to different criteria (Dillon and Pizzagalli, 2007, Friedman and Miyake, 2004, Harnishfeger, 1995, Hommel, 2011, Nigg, 2000) they all share the requirement to inhibit the processing of a prepotent, bottom-up generated, inappropriate response. Given that the inhibition of prepotent responses is thought to be mediated by prefrontal brain areas (Aron et al., 2007, Rieger et al., 2003, Rubia et al., 2003) these measures are often discussed within the framework of prefrontally mediated cognitive control (Miller & Cohen, 2001). Following Friedman and Miyake (2004) we refer to these measures as “inhibition-related functions”.
Inhibition-related functions have been found to be significantly heritable (Friedman et al., 2008, Groot et al., 2004, Heiser et al., 2006, Schachar et al., 2011). It has been shown using a variety of imaging methods and using several of the above mentioned paradigms that the prefrontal cortex, most prominently the inferior prefrontal cortex and the supplementary motor area (SMA), as well as subcortical striato-thalamic projection targets represent a neural correlate common to such paradigms (Aron et al., 2003, Aron et al., 2004, Aron et al., 2007, Eagle et al., 2008, Ettinger et al., 2008, Picton et al., 2007, Rieger et al., 2003, Rubia et al., 2007, Rubia et al., 2003).
Of importance to the clinical domain, deficits in inhibition-related functions represent a prevalent dysfunction in a number of psychiatric disorders including schizophrenia (Henik and Salo, 2004, Hepp et al., 1996, Lipszyc and Schachar, 2010), attention-deficit/hyperactivity disorder (ADHD) (Boonstra, Kooij, Oosterlaan, Sergeant, & Buitelaar, 2010) and obsessive-compulsive disorder (OCD) (Chamberlain, Fineberg, Blackwell, Robbins, & Sahakian, 2006). The study of inhibition-related functions in the clinical context is of importance for at least three reasons, namely to increase our knowledge concerning the (1) neurocognitive pathophysiology, (2) genetics and (3) pharmacological mechanisms of the psychiatric disorders mentioned above.
First, given the proximity of inhibition-related cognitive tasks to the neural level (Siebner, Callicott, Sommer, & Mattay, 2009) these have been used profitably to assess the cognitive profile and associated brain functional alterations in different psychiatric and neurological patient groups (Aron and Poldrack, 2005, Aron et al., 2003, Gauggel et al., 2004, Hutton and Ettinger, 2006). Second, on the basis of the observed heritability of inhibition-related functions (Gagne and Saudino, 2010, Kuntsi et al., 2006) it has been investigated to what extent the deficits observed in patient groups represent endophenotypes, or markers of genetic risk. For example, some studies have shown that not only patients with schizophrenia, OCD or ADHD but also their clinically unaffected biological first-degree relatives show inhibition-related deficits (Calkins et al., 2008, Chamberlain et al., 2007, Curtis et al., 2001, Ettinger et al., 2004, Ettinger et al., 2006, Schachar et al., 2005, Slaats-Willemse et al., 2003, Thaker et al., 2000) supporting the notion that such deficits share common genetic background and represent an endophenotype for these psychiatric conditions. Third, pharmacological treatment studies of psychiatric patients (Klein et al., 2002, Meere et al., 1999, Scheres et al., 2003) as well as single-dose administration studies in healthy volunteers (Barrett et al., 2004, Chamberlain et al., 2009, Costa et al., 2012, Ettinger et al., 2009, Nandam et al., 2011) have applied inhibition-related tasks as treatment targets in repeated measures designs in order to better understand the mechanisms of action of compounds such as nicotine, methylphenidate, atomoxetine, or antipsychotics at the neurocognitive level.
Overall, a large body of evidence has thus accumulated on inhibition-related (dys)-function across different neuropsychiatric conditions for these three purposes (for review, see Aron and Poldrack, 2005, Hutton and Ettinger, 2006, Lipszyc and Schachar, 2010, Westerhausen et al., 2011). Importantly, all three approaches require precise and valid measurement using reliable paradigms. Reliability is a key psychometric criterion of any psychological test (Anastasi & Urbina, 1997) and a necessary prerequisite for its validity. Reliability is most frequently measured in terms of temporal stability (or test–retest reliability) and internal consistency (e.g. Cronbach’s alpha). The issue of temporal stability is particularly important for treatment studies involving repeated assessments (Becker, 2007) and is also an explicit validity criterion of an endophenotype (Gottesman & Gould, 2003).
Therefore, our main aim was to provide a comprehensive assessment of the temporal stability and internal consistency of inhibition-related paradigms that are commonly used in the clinical and cognitive neuroscience literature. Specifically, we focussed on the go/no-go, stop-signal, antisaccade, Eriksen flanker, Simon, Stroop and CPT paradigms and studied healthy volunteers in order to obtain a clear assessment of reliability in the absence of possibly interfering symptom or treatment effects observed in clinical populations. Previous studies of individual tasks have generally reported moderate to high reliabilities (Cornblatt et al., 1988, Ettinger et al., 2003, Kindlon et al., 1995, Klein and Berg, 2001, Kuntsi et al., 2005, Logan, 1994, Smyrnis, 2008, Soreni et al., 2009, Zabel et al., 2009). Our aim was to estimate the stability of a number of widely used tasks in one study in order to allow comparison of reliability estimates across tasks.
It is known from psychometrics that the reliability of a test is positively correlated with the number of items in the test and that the reliability will increase if the scale is extended with items that do not reduce the average inter-item correlation (Carmines and Zeller, 1979, Spearman, 1910). Therefore, we explored whether such a relationship would also be observed in the paradigms under investigation here by calculating test–retest reliability coefficients not only for the total number of trials in each task but also for subscores consisting of the first 25%, 50%, and 75% of the trials to assess the adequacy of test length.
An issue related to the temporal stability of cognitive control concerns the plasticity of such functions. Effects of repeated exposure are often observed on the level of cognitive performance (Dahlin, Nyberg, Backman, & Neely, 2008); these typically represent improvements due to practice or increased task familiarity and strategy use. Importantly, such between-session changes represent a potential confound in the study of pharmacological influences on performance (Ettinger et al., 2003, Klein et al., 2002). Therefore, in this study we also aimed to characterise the extent to which performance improves from test to retest over intervals often used in clinical treatment studies, viz. approximately 11 weeks.
Importantly, plasticity effects may be observed not only from one session to the next but also within a single testing session. Such plasticity may represent learning due to rapid adaptation to the task demands or acquisition of task relevant strategies (Kelly et al., 2006, Verbruggen and Logan, 2009) as well as deterioration in performance e.g. due to fatigue. Therefore, the extent to which performance varies, improves or deteriorates over several trials within the same session was also explored in this study by studying systematic within-session changes. Finally, given the importance of intra-individual variability in performance particularly in the clinical context (see e.g. Kaiser et al., 2008, Klein et al., 2006, Kuntsi et al., 2010, Sergeant, 2005, Vaurio et al., 2009) the intra-individual standard deviation and the coefficient of variation, ICV, of performance measures were also investigated (Bellgrove et al., 2004, Klein et al., 2006, Nandam et al., 2011).
Section snippets
Participants
The present study was undertaken in a sample of healthy, non-medicated, unrelated adults. Volunteers were recruited using local newspaper advertisements, flyers and mailing lists. Ex ante participants were screened by telephone for basic inclusion criteria. At the first assessment they provided demographic and psychometric information. Inclusion criteria were: Age 18–55 years, good German language skills, no history of neurological complications, no current medical condition, and no current
Descriptive statistics
Twenty-three participants completed the study (age mean = 23.70, SD = 3.02, range 20–31; 69.6% females; years of education mean = 15.57, SD = 1.58; estimated verbal IQ mean = 114.78, SD = 9.80). The average test–retest interval was 77.65 days (SD = 25.97, range 28–105). Analysis of age effects showed that a number of variables on the Eriksen flanker, go/no-go and Simon tasks correlated significantly with age (all p < 0.01): In the Eriksen flanker task at baseline the variables mean RT (r = 0.59), SD of RT (r =
Discussion
Reliability is a key psychometric criterion for any psychological test. In this study we assessed commonly used inhibition-related paradigms by estimating test–retest reliability as a measure of temporal stability and internal consistency. To explore the relationship between test length and reliability we calculated test–retest reliability coefficients for subscores of differing lengths. Finally we investigated between-session as well as within-session changes in performance to provide an
Limitations
A first limitation of our study concerns the generalizability of the findings. In order to avoid interference of age effects, clinical symptoms or treatment effects, we examined a healthy, young sample with specific tasks. This of course limits the extent to which these findings generalise to other (e.g. older, less educated or clinical) populations and other variants of inhibition-related tasks. Therefore, the reliability measures observed in this study should be seen as sample representative
Conclusions
In summary, most of the assessed measures of inhibition-related functions displayed good test–retest-reliability and good internal consistencies combined with a lack of systematic within- and between-session changes in healthy individuals. Therefore, these measures may represent inhibition-related traits which are important for clinical research and may be used profitably in pathophysiological, genetic and pharmacological studies. Future research with variation of task parameters, sample
Acknowledgment
This work was supported by the Deutsche Forschungsgemeinschaft (ET 31/2-1).
References (116)
- et al.
The cognitive neuroscience of response inhibition: Relevance for genetic research in attention-deficit/hyperactivity disorder
Biological Psychiatry
(2005) - et al.
Inhibition and the right inferior frontal cortex
Trends in Cognitive Science
(2004) - et al.
Horse-race model simulations of the stop-signal procedure
Acta Psychologica
(2003) - et al.
Using brain imaging measures in studies of procognitive pharmacologic agents in schizophrenia: Psychometric and quality assurance considerations
Biological Psychiatry
(2011) - et al.
Practice effects and the use of alternate forms in serial neuropsychological testing
Archives of Clinical Neuropsychology
(2005) - et al.
The functional neuroanatomical correlates of response variability: Evidence from a response inhibition task
Neuropsychologia
(2004) - et al.
Eye movement dysfunction in first-degree relatives of patients with schizophrenia: A meta-analytic evaluation of candidate endophenotypes
Brain and Cognition
(2008) - et al.
Cognitive neuroscience treatment research to improve cognition in schizophrenia II: Developing imaging biomarkers to enhance treatment development for schizophrenia and related disorders
Biological Psychiatry
(2011) - et al.
Estrogen modulates inhibitory control in healthy human females: Evidence from the stop-signal paradigm
Neuroscience
(2010) - et al.
The Continuous Performance Test, identical pairs version (CPT-IP): I. New findings about sustained attention in normal families
Psychiatry Research
(1988)
Inhibition of action, thought, and emotion: A selective neurobiological review
Applied and Preventive Psychology
Influence of history on saccade countermanding performance in humans and macaque monkeys
Vision Research
Smooth pursuit and antisaccade eye movements in siblings discordant for schizophrenia
Journal of Psychiatric Research
Effects of acute nicotine on brain function in healthy smokers and non-smokers: Estimation of inter-individual response heterogeneity
Neuroimage
Primary and secondary saccades to goals defined by instructions
Vision Research
The Stroop effect in schizophrenic patients
Schizophrenia Research
The Simon effect as tool and heuristic
Acta Psychologica
Intra-individual reaction time variability in schizophrenia, depression and borderline personality disorder
Brain and Cognition
Instrumental and test–retest reliability of saccadic measures
Biological Psychology
Intra-subject variability in attention-deficit hyperactivity disorder
Biological Psychiatry
The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis
Cognitive Psychology
Methylphenidate but not atomoxetine or citalopram modulates inhibitory control and response time variability
Biological Psychiatry
Attenuation of the prepulse inhibition of the acoustic startle response within and between sessions
Biological Psychology
Mapping motor inhibition: Conjunctive brain activations across different versions of go/no-go and stop tasks
Neuroimage
Right inferior prefrontal cortex mediates response inhibition while mesial prefrontal cortex is responsible for error detection
Neuroimage
On the stability of instability: Optimising the reliability of intra-subject variability of reaction times
Personality and Individual Differences
Psychological testing
Converging evidence for a fronto-basal-ganglia network for inhibitory control of action and cognition
Journal of Neuroscience
Stop-signal inhibition disrupted by damage to right inferior frontal gyrus in humans
Nature Neuroscience
Psychologie der Persönlichkeit
CNTRICS final task selection: Executive control
Schizophrenia Bulletin
Behavioral inhibition, sustained attention, and executive functions: Constructing a unifying theory of ADHD
Psychological Bulletin
Effects of amisulpride, risperidone and chlorpromazine on auditory and visual latent inhibition, prepulse inhibition, executive function and eye movements in healthy volunteers
Journal of Psychopharmacology
Measurement and reliability: Statistical thinking considerations
Schizophrenia Bulletin
Lessons from Darwin: 21(st) century designs for clinical trials
Current Alzheimer Research
Reliability of cephalic thermal thresholds in healthy subjects
Cephalalgia
To act or not to act, that’s the problem: Primarily inhibition difficulties in adult ADHD
Neuropsychology
Further evidence for systematic reliability differences between explicit and implicit memory tests
Quarterly Journal of Experimental Psychology, Section A: Human Experimental Psychology
Reliability and validity assessment
A comparison of the reliabilities of four types of difference scores for five cognitive assessment batteries
European Journal of Psychological Assessment
Motor inhibition and cognitive flexibility in obsessive-compulsive disorder and trichotillomania
American Journal of Psychiatry
Impaired cognitive flexibility and motor inhibition in unaffected first-degree relatives of patients with obsessive-compulsive disorder
American Journal of Psychiatry
Methylphenidate effects on neural activity during response inhibition in healthy humans
Cerebral Cortex
Coefficient alpha and the internal structure of tests
Psychometrika
Saccadic disinhibition in schizophrenia patients and their first-degree biological relatives. A parametric study of the effects of increasing inhibitory load
Experimental Brain Research
Plasticity of executive functioning in young and older adults: Immediate training gains, transfer, and long-term maintenance
Psychology and Aging
Cited by (158)
The effect of task complexity on the neural network for response inhibition: An ALE meta-analysis
2024, Neuroscience and Biobehavioral ReviewsInflamed but not impulsive: Acute inflammatory cytokine response does not impact prepotent response inhibition
2023, Journal of Affective DisordersSelf-control training supplementing inpatient multidisciplinary obesity treatment in children and adolescents
2023, Behaviour Research and TherapyThe role of inhibitory control in sport performance: Systematic review and meta-analysis in stop-signal paradigm
2023, Neuroscience and Biobehavioral ReviewsNeurobehavioral maturation of motor response inhibition in adolescence – A narrative review
2022, Neuroscience and Biobehavioral ReviewsCitation Excerpt :Dedicated studies exploring reliability of executive tasks in adult samples have found variable reliabilities for AST measures. In a test-retest study of 23 adults over an interval of 23–105 days, antisaccade latency and accuracy showed excellent internal consistency at baseline (Cronbach’s α = 0.92 and 0.94, respectively) and retest (Cronbach’s α = 0.97 and 0.94, respectively; Wöstmann et al., 2013). Test-retest reliability in the same sample showed excellent reliability for AS latency and accuracy (ICC = 0.88 and 0.92, respectively).