Test-Retest Reliability of a Computerized Neuropsychological Test Battery: A Cross-Sectional Study Assessing Cognition in Healthy Young and Old Adults, and Stroke Survivors

Objective: Dual-task methodology is widely utilized to measure sharing of attentional resources between motor and cognitive functions. Computerized neuropsychological testing is an advanced approach of cognitive assessment. The traditional paper-and-pencil tests, although are valid and reliable, have certain drawbacks hence limiting their overall usability, particularly in clinical settings. This study aimed to establish the testretest reliability of a computerized, custom-designed cognitive test battery in healthy young and older adults and chronic stroke survivors. Methods: Healthy young (n=15) and older (n=15) adults and adult chronic stroke survivors (n=15) were tested for the following domains via DirectRTTM, Empirisoft visuo-motor function (Spot & Click, SC) 1. associative memory, via a paired associative learning task (Number & Position, NP) 2. semantic fluency, (Category Naming, Cat.N) 3. information processing speed/response inhibition (Color Naming, CN) 4. Discriminant Decision Making (Unveil the Star, US) 5. visual working memory using 1-back and 2-back tasks (Triangle Tracking, TT). The outcome variables consisted 6. of reaction time, accuracy, total time of completion and total number of responses. Results: A good to excellent reliability for all the six tasks (p<0.05) for each of the three groups was observed. Results demonstrate, that stroke survivors performed worse than healthy young and older adults whereas both the healthy young and older adults showed similar performance across different tests. Conclusion: Results indicated that these computerized cognitive measures were highly reproducible and reliable. Such testing could be easily implemented by clinicians for assessing cognition and incorporated in dual-task testing and training paradigms. DOI: 10.29011/2574-7762. 000007 Citation: Jinal V, Rini V, Victor P, Sara W, Tanvi B (2017) Test-Retest Reliability of a Computerized Neuropsychological Test Battery: A Cross-Sectional Study Assessing Cognition in Healthy Young and Old Adults, and Stroke Survivors. J Psychiatry Cogn Behav 2017: J107. 2 Volume 2017; Issue 02


Introduction
Neurocognitive decline is a growing healthcare concern that can result in loss of functional independence in healthy aging baby boomers and individuals with neurological diagnoses [1].More than 5.1 million individuals in the United States are estimated to be living with cognitive impairment [2].The medical costs for older adults with Mild Cognitive Impairment (MCI) are 44% higher than those for non-impaired older adults [3].Due to structural changes in the prefrontal cortex parietal and medial temporal regions including hippocampus, cognitive functions most commonly affected in these individuals include executive function, information processing, and working and associative memory [4].In addition to age-related cognitive changes, individuals with long-standing neurologic disorders, such as stroke, may retain permanent damage to brain areas affecting cognitive, sensory, and motor functions [5].
Cross-sectional studies show that 64% of stroke survivors have cognitive impairments unrelated to dementia [6].Furthermore, among individuals presenting with MCI post-stroke, the chances of reversibility of cognitive impairment decline with the chronicity of the condition [7,8].Recent findings suggest that higher cortical centers may contribute to poor motor control during performance of daily functional motor activities like walking, grocery shopping, and driving by sharing the same attentional resources with cognitive functions such as working memory, spatial and discriminant decision making and information-processing [9,10].This has been observed through dual-task paradigms focusing on simultaneous performance of a motor and a cognitive task.Studies have shown that dual-tasking results in deterioration of performance in either motor and/or cognitive tasks, which is described as Cognitive Motor Inference (CMI) [11].There are various potential patterns of interference described by Plummer et al.(2013) namely No interference, where the performance on either task does 1.
not change relative to single-task performance.
formance remains stable whereas motor performance deteriorates.
mance remains stable and the cognitive performance deteriorates.
Mutual interference, where performance on both tasks dete-4.
riorates [12,13].CMI result from competing demands of both motor and cognitive tasks for accessing the limited and probably shared processing resources within the brain [14].
In older adults, it has been observed that motor performance is significantly altered with concurrent performance of a secondary cognitive task (dual-task) such as increased postural sway and decreased gait speed and step lengths [15,16].Such findings of deterioration in gait and balance performance are evident in stroke survivors, as well.Dual task assessments could thus provide a platform to understand the interaction between cognitive and motor systems in healthy and neurologically-affected individuals [12].
Although there is evidence suggesting a critical role of cognition in facilitating motor performance, cognitive testing and interventions have not been routinely utilized in clinical rehabilitation settings.Studies indicate that neural plasticity endures across the lifespan, and continuous cognitive stimulation is important for enhancement and maintenance of cognitive functioning [17].However, prior to providing cognitive or cognitive-motor interventions, an accurate and comprehensive neuropsychological assessment is fundamental to identify specific cognitive domains most severely impacted by aging and/or neurologic processes.
Conventional paper-and-pencil measures used to assess cognitive function tend to be lengthy, require specialized training, and are prone to manual error.They are also not feasible for use in cognitive-motor dual task paradigms.Computerized Cognition Testing (CCT) is gaining popularity and offers an alternative to some of the traditional paper-and-pencil testing measures.CCT has potential advantages over conventional testing methods as it renders precise stimulus control, consistency in administration and scoring, visually appealing interfaces, cost efficiency, ability to develop large and accurate databases and ease of administration of cognitive-motor testing paradigms [18].Computerized cognitive testing also helps to assess and monitor cognition in large sample size [19].However most of the commercially available neuropsychological test batteries are generally lengthy and require several sessions of testing as they assess multiple domains of cognition thoroughly [20][21][22][23].Without understanding the burden the patients undergo which may eventually affect the test outcome measures.Keeping this in mind and after extensive research we developed a customized computerized cognitive test battery consisting of six task assessing the six-main domain of cognition namely, visuomotor function, verbal fluency, executive function, discriminant decision making memory, working memory and associative memory and the average total time to complete this battery was 40 minutes.We choose to evaluate the test battery administered via the Direct RT Empirisoft™ over other commercially available tests as it is affordable, user friendly, allows to develop customized test stimuli and easy to modulate tests with increasing difficult levels and allow subsequent interpretation of a variety of cognitive tests without the need for specialized training.
Nonetheless, there can be certain challenges associated with use of computers to assess cognitive function in older or poststroke population.Hence a good interface design and simple cognitive tests were created that could improve the usability of computerized testing in the above-mentioned population and in turn increase the reliability and validity of the cognitive data [24,25].In a previous study, we examined the reliability and concurrent validity of a computerized cognitive test battery administered using Direct RT™ Empirisoft among healthy young adults [26].While the previous study reported positive findings, we concurred that the battery's reliability needs to be examined in other populations prone to cognitive decline.
The primary aim of this study was to establish the test-retest reliability of a custom-designed, Computerized Cognitive Test battery (CCT) in post-stroke survivors, healthy older adults and young adults, using the commercially available Direct RT™ Empirisoft, to determine its potential for use in clinical or research settings.The secondary aim was to understand the effect of aging and neurologic disorder due to stroke on cognitive functioning amongst healthy older individuals and stroke survivors.For this aim, we hypothesized that cognition would be significantly impaired in the participants with stroke as compared to healthy older adults and young adults.

Participants
Fifteen adults with chronic hemiparetic motor stroke as confirmed by the physician (age: 58.9± 6.9, years of education: 14.2± 2.7), fifteen healthy older adults (age: 64.4±5.1, years of education: 16.2± 2.3), and fifteen healthy young adults (age: 23.86 ± 1.35, years of education: 16.8 ± 1.47) participated in this study.Participants were recruited via informational flyers posted across the University of Illinois at Chicago Medical campus.Exclusion criteria for participants with chronic stroke was severe receptive or expressive dysphasia, which would limit their participation in the assessment as determined via the scores obtained on Mississippi Aphasia Screening Test [27].Individuals under medications impacting cognition or having comorbid psychiatric and medical conditions were also excluded.Healthy older and young adults were included if they had no self-reported neurological disorder or injury that could affect sensory, motor or cognitive functioning.The study was approved by the Institutional Review Board of the University of Illinois at Chicago, and informed consent was obtained from all participants.
Prior to testing, individuals premorbid IQ score was assessed using the WRAT-4 reading test (total score out of 70) after which participants completed the computerized neurocognitive test battery administered using DirectRT™, Empirisoft [28].
All testing was conducted in a silent room in order to avoid any external disturbances.For computerized testing, a screen was placed in front of the participant and a headset with microphone was used to record the responses.Each test was preceded by an instruction slide and instructions were also repeated by the examiner, after which the participant was asked to press a key when he or she was ready to start the test.Each participant underwent two testing sessions separated by 10-12 days' interval.During each session, all the tests from the computerized cognitive test battery were administered in a randomized order i.e. with respect to the order of the tests administered.

Neuropsychological Test Battery
Six domains of cognition (visuo-motor function, verbal fluency, executive function, discriminant decision making memory, working memory and associative memory) were included in the testing protocol.Visuo-motor function was assessed by measuring the response latencies after the stimulus is presented to the examinee i.e. the Spot & Click task.Both, Simple Reaction Time (SRT) and Choice Reaction Time (CRT) was assessed.The participant was presented with a stimulus (yellow circle) and asked to respond by pressing the corresponding key on the number pad depending on the position of the stimulus.Measurement of executive function consisted of the classic strop paradigm, which measures processing speed and response inhibition.The computerized version of the test (Color Naming) consisted of two conditions.In the first condition the participant was asked to read the words that were printed in the same color ink (congruent condition).Condition 2 required the participant to name the ink (color) in which the color word was printed.The actual word frequently differed from the ink in which it was printed (incongruent condition).The total time taken to complete the task was recorded.Unveil the Star task assessed the manipulation and retention of visuo-spatial information and discriminant decision making.In this task, the participant was required to search for a blue star in multiple red boxes without clicking on the same box twice where the star was previously found.There were three increasing levels-of-difficulty as the task progressed.The total time taken to complete the task, along with the total number of errors was recorded.Test of verbal fluency included the Category Naming task in which participants' semantic fluency was assessed.Participants were provided a category cue (i.e.animals, boys' names, or countries) and were given one minute to provide as many words as possible relating to that category.They were instructed to avoid repeating the same responses.The voice responses were recorded by the computer via the microphones.Working memory was also assessed using the Triangle Tracking task in which the participant was presented with a sequence of stimuli and asked to indicate when the current stimulus matches the one from n-steps earlier in the sequence.We used 1 and 2 steps earlier for the current protocol.Lastly, the Number & Position task was administered to assess associative memory, where the participant was presented with a slide displaying numbers in a grid and the participant was asked to memorize the position of the number (i.e.7-center, 3-Upper left, 1-lower right) after which a slide with the single number in the center was presented and the participant was asked to recall the position of the number from the previously presented slide.The outcome variables for each of the computerized neurocognitive test battery are presented in Table 1.

Statistical Analyses
Descriptive statistics (mean ± SD) were performed for all cognitive test variables.Intra-Class Correlation Coefficients (ICC) was used to determine the reliability of each of the computerized tests in the cognitive battery.Test-retest reliability was characterized as excellent (ICC >0.8), good (ICC 0.6-0.79),moderate (ICC 0.4-0.59),fair (ICC 0.2-0.39)and poor (ICC<0.2) [29].The level of agreement between the two testing sessions is represented using Bland Altman plots.One-sample t-test was used to analyze whether the bias (difference) between the mean score of the two sessions was significantly different from zero.ANCOVA was performed to analyze the significant differences between the groups in cognitive test variables using the premorbid IQ score and Years of education as covariates among the three groups followed by posthoc Tukey's test.The statistical significance was set at p<0.05.All statistical analyses were performed using Statistical Package for Social Sciences (SPSS), version 22.0 (Chicago,IL version 22).

Demographic data
Demographic comparisons between the three groups (i.e., stroke, healthy older adults, and healthy young adults) are shown in Table2 together with the neurological data.* Difference in means between stroke and Healthy older adults statistically significant (p < 0.05).
# Difference in means between stroke and Healthy young statistically significant (p < 0.05).

Table 2:
Demographics and disease characteristics of chronic stroke patients, older adults and young adults.

Reliability
Paired t-tests showed no significant difference between the two testing sessions among the different variables.Test-retest reliability data revealed significant Intra-Class Correlation (ICC) with 95% CI for each of the Plots which displays the level of agreement between the testing sessions in Figure 1.The results of the ANCOVA on the Computerized Cognitive Test (CCT) battery showed that the difference in the cognitive test between the stroke group, the healthy older adults group, and the healthy young adults group were statistically significant (p<0.05), as presented in Table 4. Premorbid IQ scores and years of education were significant covariates for the Reaction Time test variables.Tukey's HSD test revealed significant differences in the CCTs between the three groups.The stroke group showed a significantly longer reaction time on Spot & Click task, longer time for completion for Color Naming task than the healthy older group (p< 0.05) and healthy young group (p< 0.05).The stroke group presented significant more errors for Unveil the Star task as compared to healthy young group (p<0.05).The stroke group presented significantly lower number of correct responses and decreased accuracy for the Triangle Tracking task and Number & Position task as compared to the healthy older adult group (p<0.05) and healthy young group (p<0.05).The healthy older adult group showed a significantly longer reaction time for the Spot and Click task and an increased time for completion for the Unveil the Star task, as compared to the healthy young adult group (p<0.05) as presented in Table 5 and Figure 2 (-23.5 --1.7) (-24.9 --3.4) (-9.1 -12.4) Table 5: Computer generated test scores for stroke patients, healthy older adults, and healthy young adults reporting mean differences with 95% CI.

Discussion
Overall, the findings from our study indicate that the Computerized Cognitive Tests (CCTs) administered via Direct RT™ Empiri soft are moderately reliable measure of cognitive functioning across two testing sessions in chronic stroke survivors, healthy older adults and young adults.Further, the secondary finding of our study was that the CCTs were able to detect difference in cognition between a small sample of stroke survivors, healthy older adults and young adults.

Test-retest reliability
The test-retest reliability correlations measured with Intra-Class Correlation Coefficient (ICC) for all of the computerized cognitive tasks assessed across two testing sessions ranged from 0.75-0.98 for all of the groups together.On further analyzing the reliability coefficient for individual groups, a high test-retest reliability coefficient was obtained for the young and older adults as compared to chronic stroke survivors for both simple and choice reaction task.The range of the reliability coefficients in this study across all the groups for reaction time was consistent with the previously reported ranges i.e.ICC=0.60-0.98 [30-32].
Furthermore, the high test-retest coefficients reported in the study were not biased depending on the practice effects because the stimuli across both testing sessions were unpredictable for the participants being tested as the stimuli presented were random.It has been proposed by Lowe and Rabbit [30] that reaction time is a non-strategic task and therefore might not result in forming a memory as to when the stimuli will appear.This task can thus yield high test-retest correlation, suggesting that the likelihood of practice effects was reduced as a result of unexpected stimuli.
For the Category Naming task (verbal fluency), a high testretest reliability coefficient for all groups together i.e.ICC=0.89 and for individual groups was attained.The Unveil the Star task (Discriminant Decision Making) and Color Naming task (pro-cessing speed/response inhibition) demonstrated high ICCs for the groups together.Moderate test-retest correlation for total time taken to complete Unveil the Star task was observed for individual groups.Examination of the data suggests that participants required less time to complete the task during the second session as compared to the first session possibly due to familiarity with the test.Similar results were obtained for the Triangle Tracking task (i.e., working memory) and the Number & Position task (i.e., associative memory) wherein moderate test-retest reliability was observed for all three groups together.Hence, we suggest that working memory tasks are usually strategy-driven, and accurate results in order to interpret test-retest reliability scores can be provided only when they are novel as performance on them can improve once the participant discovers an optimal strategy for it [30] or alternatively the stimuli presented in the some of the memory tasks were learned and remembered between the sessions resulting into moderate testretest reliability for all three groups together [33].

Group Differences in cognitive abilities
Our secondary findings focused on understanding mean group differences and interpreting which cognitive domain is primarily impaired post-stroke.The reaction time recorded for the choice reaction time task and the time difference calculated between simple and choice reaction time tasks (CRT-SRT) demonstrated higher means for stroke survivors compared to healthy older and young adults.A significant difference in the reaction time was observed between the older adults and young adults.Reaction time has been extensively studied in the literature and is found to increase with age [34,35].It has also been suggested by Brein et al that reaction time is one of the most sensitive markers of structural and functional deterioration in the aging central nervous system which is further hampered if there is underlying neurological insult such as stroke [36].
The stroke group demonstrated greater number of errors for Unveil the Star task (discriminant decision making) as compared to the young adults.They also demonstrated decreased accuracy on the Triangle Tracking task (working memory) and the Number & Position task (associative memory) as compared to older adults and young adults.For Unveil the Star task, the older adults and stroke groups required more time to complete the task as compared to young adults.One explanation for this result can be related to the increased complexity inherent in the task, as the individual is required to follow strategic sequence in which he/ she successfully avoids the boxes on which they have already clicked.Further we may suggest that stroke affecting prefrontal and frontal areas could have impacted processing within the working memory system, thereby requiring more time and have less accuracy in completing the task but more information with respect to the site of the lesion and affected lobe is warranted.
Studies have suggested that executive function tasks could be as equally complex as the working memory tasks, demanding greater attentional resources for information processing and response inhibition by suppressing task-irrelevant information [9,37] resulting in increased time of completion for the incongruent slide of the Color Naming task as compared to the congruent slide.For the Category Naming task, in which the participants were asked to recall words belonging to a particular semantic category (e.g.animal), significant difference for number of correct responses was observed between chronic stroke survivors and healthy older and young adults.The stroke group demonstrated decreased number of correct responses for the Category Naming task (semantic fluency) as compared to the other two groups.This suggest that that verbal fluency is a frontally mediated cognitive function and hemiparetic stroke can result in poor performance on this task [38].
Notwithstanding these results, the study had certain limitations.The overall sample size was small, and reliability measurements should be repeated in a larger sample and also try to limit practice effects due to familiarity to the paradigm.Furthermore, for future implication we would like to incorporate not only the type but also the site of lesion for a better understanding of which cognitive function is affected more by comparing the sites of the cerebral lesion.Also, the CCTs can be further explored in people suffering from MCI, dementia or any other cognitive impairments.
To conclude, the results from the study suggest that the customized computerized cognitive test battery is a reliable tool to assess cognitive function.The custom developed CCT'S may be a good tool to capture more subtle changes in cognitive functioning, relative to paper-and-pencil clinical global cognition tests such as the Montreal Cognitive Assessment which can be useful as screening tool [39].Computerized measures also require less training for administration and can be employed in busy clinical settings without undue burden on staff.Computerized batteries can be used as either screening instruments to identify patients requiring further evaluation by a neuropsychologist, or as part of a standard neuropsychological assessment.Lastly, CCTT's can be employed as dual-task paradigms for cognitive-motor rehabilitation in clinical settings.Future studies should establish sensitivity and specificity of this CCT for identifying Mild Cognitive Impairment and dementia among patients with stroke and other neurological problems.

Figure 1 :
Figure 1: Bland-Altman plots displaying limits of agreement: mean (solid line) ±1.96 SD (dotted line) of the difference between test session 1 and test session 2 values for the variables: (a) number of correct responses; (b & f) total time completion; (c) total errors (d) choice reaction time (CRT); and (e) accuracy (%).(The represents the stroke group, represents healthy-Group Effects

Figure 2 :
Figure 2: Plots displaying means of each outcome variables for test session 1 and 2 which are not significantly different from each other (p >0.05).a) Choice reaction time b) Total time completion for Color Naming task-incongruent slide c) Unveil the Star task respectively.d) Accuracy (%) for Number & Position task e) Accuracy (%) for Triangle Tracking (2-back) f) number of correct responses for the category naming task.Difference in means with significance level (p ≤ 0.05) † = Difference in means between stroke and healthy older adults.# = Difference in means between stroke and healthy young adults.* = Difference in means between healthy older adults and healthy young adults.

Table 1 :
Computerized cognition tests assess for test-retest reliability and outcome variables.

Table 4 :
Computer generated test scores for stroke patients, healthy older adults, and healthy young adults. .