Participants

Neurologically healthy individuals (NHI) group.

One hundred and six native speakers of Russian without history of neurological or psychiatric disorders or substance abuse participated in the study (see Table 1 for participants’ demographic characteristics). The level of formal education varied from secondary school (typically 10 years in Russia) to a university degree (typically 15 years), with the majority of the participants (80.2%) having some form of higher education degree (12–15 years). Most participants were right-handers (N = 101); three were left-handers and two were retrained left-handers. All participants reported normal or corrected-to-normal vision and hearing acuity.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Participants’ demographic characteristics.

https://doi.org/10.1371/journal.pone.0258946.t001

Main PWA group.

All PWA except one were recruited at the Center for Speech Pathology and Neurorehabilitation, Moscow, Russia. None of them had history of neurodegenerative disorders or substance abuse. There were 85 participants in the PWA group (see Table 1 for demographic characteristics); 44 participants were in the young (age: M = 49.1, SD = 10.3) and 41 in the elderly age cohorts (age: M = 66.8, SD = 4.8). The level of formal education varied from incomplete secondary school (8 years) to a university degree (typically 15 years), with the majority of the participants (58.8%) having some form of higher education degree (12–15 years). Most participants were premorbidly right-handers (N = 80), one was left-handed, one–a retrained left-hander, and three were ambidextrous.

All the PWA had speech and language deficits due to focal brain damage of various etiology confirmed by a CT or an MRI scan. Most participants had damage to the left hemisphere (N = 82); two had bilateral damage; one had combined damage to the left hemisphere and the vertebrobasilar area. The majority of the PWA (N = 77) had a single or recurrent stroke (ischemic or hemorrhagic); six had traumatic brain injury (TBI); one had impairments due to tumor resection (meningioma); and one had a complex etiology (TBI + infection + toxic). All were diagnosed with aphasia after a standard clinical examination by a certified speech pathologist or neuropsychologist. According to medical histories, all participants had normal or corrected-to-normal vision; seven participants had decreased hearing acuity.

Participants with aphasia also had a range of concomitant impairments, including deficits in speed of processing, attention, working memory, and motor control as indicated in their medical histories. All of these noted cognitive deficits were secondary and minor relative to their primary aphasia diagnosis and these deficits did not compromise informed consent, the validity of the assessment or the interpretation of outcomes. No patients in the current sample had visual agnosia or neglect, one participant had hemianopsia, however, this was accommodated during testing by placing the stimuli in the preserved field of view. For generalizability purposes in this project, it was important to have a varied PWA sample, representative of the target population in a real-life clinical setting. Accordingly, we opted for maximally broad inclusion criteria, without compromising validity of findings.

Depending on the type of aphasia the patient was classified as either having a fluent or a non-fluent type of aphasia, or determined non-classifiable in the cases when the type of aphasia could not be identified unambiguously (see Table 2) according to the aphasia classification by Luria [3, 4, 30]. Specifically, individuals diagnosed with motor (roughly corresponding to Broca’s aphasia in the Boston classification) and/or dynamic (transcortical motor) aphasia based on the comprehensive neuropsychological evaluation were grouped into the “non-fluent” group, while individuals with sensory (Wernicke’s) and/or acoustic-mnestic (anomic) aphasia were included in the “fluent” group (see [3, 4, 30] for a more detailed discussion of the correspondence between various aphasia types in the different classification systems). Here we would like to emphasize that the assignment of the “non-fluent” and “fluent” labels was not grounded on the performance on a particular fluency task, but on the qualitative division between two major aphasia syndromes. This distinction has been acknowledged in many different aphasia classifications and remains widely accepted [31].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Fluency groups determined according to Luria’s classification of aphasia in the main PWA group.

https://doi.org/10.1371/journal.pone.0258946.t002

Additionally, for all the PWA except one individual a clinically established comprehensive language battery, the ASA [5] (described in detail in the Introduction) was also administered, which provided an overall score reflecting general severity of language impairment.

Rationale and development of the RAT

General structure of the test.

The RAT is composed of three main parts: auditory comprehension, repetition, and oral production. Within each domain, individual subtests target processing at different language levels including phonological, lexical-semantic, syntactic, and discourse. Auditory comprehension is assessed at different levels of linguistic analysis with the following subtests: nonword discrimination, auditory lexical decision, single word comprehension of nouns and verbs, sentence comprehension, and discourse comprehension. This hierarchical set of subtests allows to determine the language level(s) at which comprehension breaks down. The repetition subtests include: nonword, word, and sentence repetition–and are designed to evaluate integrity of sublexical and lexical pathways for repetition, along with auditory short-term memory capacity. The oral production subtests encompass naming of objects and actions, sentence production with syntactic priming, and picture-elicited discourse production. This selection of subtests helps to distinguish between impairments in production at the lexical-semantic and syntactic levels, and to assess these abilities cumulatively under relatively natural conditions (in discourse production), enabling detection of mild residual deficits in expressive language abilities.

Within each subtest, (psycho)linguistic variables known to impact language processing in PWA are systematically manipulated to ensure inclusion of items of varying difficulty and to provide detailed information about intact and impaired components of the language system. Where applicable, item selection was based on relevant psycholinguistic parameters of the verbal and pictorial stimuli (imageability, age of acquisition, name agreement, image agreement, object / action familiarity, visual complexity: http://en.stim-database.ru, [32–34]; lemma frequency [35]). The manipulation of critical (psycho)linguistic variables helps to dissociate PWA’s difficulties at specific linguistic levels more accurately. Additionally, matched stimuli across some of the subtests allow for a direct comparison of PWA’s performance across tasks and domains. Specifically, the items in the single word comprehension subtests were matched with the items in the naming subtests on a number of psycholinguistic parameters. Additionally, the sentence comprehension and production subtests employed similar syntactic constructions. See Table 3 for a brief description of the subtests’ design and supporting information for a comprehensive account and a detailed rationale behind their construction (see S1 File).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Description of the RAT subtests.

https://doi.org/10.1371/journal.pone.0258946.t003

All visual stimuli accompanying different RAT subtests consisted of black-and-white line drawings. For single-word comprehension and naming subtests, pictures were taken from the Database of Russian Verbs and Nouns [32–34]. The visual stimuli for the sentence comprehension and production subtests, and the picture for the discourse production subtest were specially created for the RAT by the same artist. Auditory stimuli for all the subtests were recorded in a studio by a professional male speaker.

Administration of the RAT

The RAT was presented on the Samsung Galaxy Tab A (model SM-T585) or Tab 4 (model SM-T531) tablets. Both tablet models had a screen size of 10.1 inches and the following screen resolution: 1920 x 1200 and 1280 x 800 pixels, respectively. While the test can be administered and scored in a traditional paper-and-pencil format (a paper-copy version of the stimulus cards and the scoresheets in Russian are available at– https://www.hse.ru/en/neuroling/research/rat/), the reported data were collected using the electronic version of the test. No special tablet use skills, such as dragging or swiping, are required on the participant’s part to complete the tablet-based version of the RAT, and all navigation between subtests and items (except in the comprehension subtests, see below) was done by the examiner. The whole test took 1–2 hours to administer (depending on the patient’s aphasia severity). The NHI group completed the test in one session; the PWA required 1–3 testing sessions (depending on the fatigue levels) within the same week to complete the test.

All the subtests started with presentation of written instructions on the tablet screen (see S1 File for individual subtest instructions). The examiner read them aloud and provided additional explanations if necessary. The instructions were followed by several practice items (typically three). Here, the examiner ensured that the participant understood all the instructions, repeating the practice items multiple times and providing feedback and clarifications, if needed. After clarifying all the questions, the actual test items were presented. For all the subtests, the visual stimulus appeared first and was followed (where applicable) by the auditory stimulus with a 2-second delay. For the comprehension subtests, the next trial was automatically triggered upon response selection. For the repetition and production subtests, the examiner manually advanced to the next trial once the participant provided a verbal response. Participants’ responses on the repetition and production subtests were not time-limited, although the examiner urged the participants to provide a response in case of an abnormally prolonged hesitation, i.e., when there was a pronounced delay in responding to a test item relative to responses provided previously by the same patient. If they failed to do so, the examiner proceeded to the next item and the item was marked as no response (incorrect). A single repetition of the test item upon request was allowed and incurred no penalty. No meaningful cues from the examiner were permitted. Sometimes for psychological reasons (e.g., to minimize a participant’s frustration), it was necessary to repeat the stimulus item multiple times or provide a cue; however, subsequent responses were not scored, and the item was marked as incorrect. In cases when the patient gave continually erroneous responses, the examiner was instructed to still complete administration of the subtest. Only if the patient persistently, for several items in a row provided no responses, the remaining items were marked as incorrect.

Scoring guidelines

For the comprehension subtests, each response was automatically registered as correct or incorrect. As stated above, a single repetition of the item was allowed. The correct items gained a score of 1, incorrect–a score of 0; the total raw score for each subtest was the sum of the item scores.

In the current study, the recorded verbal responses in the production and repetition subtests were downloaded from the tablet to a computer, transcribed and scored manually according to subtest specific criteria (see below). In terms of real-life procedures, clinicians can fully rely on the application for scoring by replaying the individual responses on a trial-by-trial basis on the tablet after the test is administered and scoring them there. Alternatively, they can perform the scoring in real time and score responses while the test is being administered in the paper protocol. In either case, a full transcription is unnecessary for clinical administration of the test. Self-corrections in the repetition and production subtests entailed no penalty, but only the last verbal response was scored (even if the original response was correct). Additionally, verbal responses were not marked down for typical dysarthric distortions.

In the nonword and the single word repetition subtests, a score of 1 was given for a correctly repeated item (nonword/word), 0.5 for a phonological paraphasia (when more than 50% of the target word is spared, and the target is still recognizable), and 0 for all other types of errors. In the sentence repetition subtest, each word in the sentence was scored separately in a similar fashion. For each correctly repeated content and functional word in the sentence, 1 point is given. Phonological paraphasias and word form errors are scored as 0.5, while omissions and other errors are given a score of 0. Word order changes, repetitions, omissions, and insertions that altered the word order in the sentence incur an order penalty of 1 (irrespective of the number of such errors). Then the score for each sentence is calculated as the sum of the word points minus the order penalty. The total raw score for each subtest was the sum of the item scores.

In the naming subtest, a correct response was scored as 1, and all error types as 0. In the sentence production subtest, each sentence was evaluated according to four criteria: consistency with the prime, grammaticality, lexical-semantic adequacy, and other aspects of phrase appropriateness. A score of 1 was given if a criterion was met, and 0 if it was not. The sum of the scores on the four criteria constituted the score for a particular item; the total raw score was the sum of the item scores. Performance on the discourse production subtest was rated on a 5-point scale (with higher score corresponding to better performance) on each of the four criteria: fluency, grammatical complexity, paraphasias, and informational content. The total raw score for this subtest was the sum of the scores on the four scales.

Finally, subtest accuracy was defined as the percentage of correct responses out of the maximum possible score for all the scored items. As a measure of overall language impairment, we computed the General Aphasia Quotient (GAQ). It is used to determine presence and severity of aphasia and is calculated as an average of total percentage scores for all the subtests. Thus, it ranges from 0 to 100%. GAQ can be calculated only if total scores of all the subtests are available (i.e., all the subtests were administered to the participant, fully or partially).

More comprehensive explanation of the scoring procedures for each subtest are provided as supporting information in the detailed description of the RAT subtests (see S1 File). Prospective users of the RAT are encouraged to follow the same administration and scoring guidelines to ensure the validity of results.

Data analysis

We established the following psychometric properties of the test based on the NHI and PWA data (all statistical analyses were done in R [36] and figures were drawn in ggplot2, ver. 3.3.2 [37]):

1. Cut-off values indicating impairment for both individual subtests and the overall score (GAQ) were determined. For individual subtests, the 5^th percentile of the respective control group was used as the cutoff. To determine a cut-off value for the presence of aphasia, the Receiver Operating Characteristic (ROC) curve analysis was performed using the GAQ of the NHI and the PWA groups. These analyses support the use of the test to identify a language impairment in individuals with brain damage relative to healthy controls.
2. Severity ranks (mild, moderate, severe) for each subtest and GAQ were established based on percentile ranges of the PWA group. This analysis offers evidence that the RAT can measure impairments of varying severity in different domains and the test can be used to quantify overall aphasia severity.
3. Concurrent validity of the test was ascertained by correlating the GAQ based on the RAT with the overall scores on the ASA [5]. This further supports the claim that the RAT can capture the full spectrum of aphasic language disorders.
4. Construct validity was evaluated based on correlation patterns between the RAT subtests. These findings provide further support that the test offers a multidimensional evaluation of linguistic strengths and weaknesses across multiple language domains.
5. Inter-rater and test-retest reliability of the test was determined using inter-class correlations. These estimates offer evidence that the RAT measures language deficits reliably over time and irrespective of the rater, which is important for all proposed applications of the test.

Overall performance of the NHI and PWA groups

Descriptive statistics for each subtest scores and GAQ for the NHI and PWA for both age cohorts are provided as supporting information (see S1 Table). Additionally, accuracy scores for each subtest for the two participant groups across the two age cohorts are presented as boxplots in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Accuracy subtest scores and the General Aphasia Quotient (GAQ) for the control group of neurologically healthy individuals (NHI) and the main group of people with aphasia (PWA) for each age cohort.

The box represents the interquartile range, with the central line marking the median. The whiskers denote the largest/smallest values within 1.5 times the interquartile range above/below the 75^th/25^th percentile. Values falling outside of that range are shown as black points. Red lines indicate cutoff thresholds: dashed for each subtest, representing the 5^th percentile of the control group, and solid for the GAQ cutoff, determined according to Receiver Operating Characteristic (ROC) curve analysis. The cyan box with percentages represents percent of PWA performing in the impaired range (at or below cutoff for normal performance for a given subtest).

https://doi.org/10.1371/journal.pone.0258946.g001

The distribution of scores clearly demonstrates that the NHI group performed near ceiling in most subtests, apart from Discourse Comprehension, Discourse Production, and Sentence Production. This lack of a ceiling effect in these specific subtests could be due to the complexity of these tasks and participants’ variability in non-linguistic pragmatic skills (Discourse Comprehension, Discourse Production) and memory (Discourse Comprehension, the priming component of Sentence Production). Additionally, the two age cohorts in the NHI group performed comparably, except for subtests targeting phonological processing and/or memory (independent-samples two-tailed Welch t-tests: Discourse Comprehension, t = -3.47, p = .001; Nonword Repetition, t = -4.17, p < .001; Sentence Production, t = -4.01, p < .001; and borderline significant for Sentence Repetition, t = -3.01, p = .0039), with significance value Bonferroni-corrected to p = .05/13 = .0038. This reflects typical patterns observed in the aging population due to increasing sensory deficits, declining working memory and general slowing [38, 39].

Performance of individuals with aphasia was more variable, especially for repetition and production subtests, reflecting different language profiles and deficit severity in the main PWA group (Fig 1). Out of the whole aphasia group (N = 85), 68% (N = 64) completed all the subtests of the RAT.

As expected, PWA obtained significantly lower scores across all subtests compared to the NHI group in both age cohorts (based on independent-samples two-tailed Welch t-tests with significance value Bonferroni-corrected to p = .05/26 = .0019). Two exceptions were Noun Comprehension in the young group and Lexical Decision in the elderly group, where the difference trended towards significance (p = .003 in both cases). To determine the effect of fluency, an independent-samples two-tailed Welch t-test was performed between fluent (N = 36) and non-fluent groups (N = 36). No significant differences (p > .05) in any subtest were revealed, underscoring the prevalence of various linguistic deficits irrespective of aphasia group and the importance of detailed linguistic diagnostics.

Additionally, we wanted to verify that documented decreased hearing acuity in PWA was adequately corrected and that it was not influencing performance on subtests where peripheral auditory function is particularly critical, such as the Nonword Discrimination, Lexical Decision, and Single word Comprehension. We compared performance on those subtests between PWA with reported hearing acuity deficits (N = 7) and without (N = 78) using the Fischer’s exact test that allowed to establish whether the distribution above and below cutoff was similar in the two groups. Across all the four subtests of interest no significant differences between the two groups in the proportion of patients performing below the cutoff were observed verifying that hearing deficits did not influence performance on the subtests.

Item analysis

We calculated item passing rates (average score for each item) based on the PWA data (see S3 Table for individual item passing rates for all subtest). Comprehension subtests had overall higher item passing rates (mean of item passing rates across comprehension subtests ranged between 0.77 and 0.94) compared to both repetition (0.51–0.75) and production (0.44–0.64) subtests, again reflecting the fact that subtests targeting expressive language abilities are on average more difficult for PWA. However, all subtests, except for Noun and Verb Comprehension subtests, comprised items sufficiently ranging in difficulty. Relative to other subtests, the Noun and Verb Comprehension subtests did show a narrower spread in item passing rates and generally higher values, signifying that single word comprehension abilities are the least impaired in aphasia.

Next, we calculated corrected item-total correlations (correlation between the item score and the total subtest score minus that item, computed using the psych package for R [40]) based on the PWA data (see S4 Table for individual item values). The high values (mean of corrected item-total correlations across subtests ranged between 0.36 and 0.86) and the observed spread reflected good overall coherence and at the same time sufficient discriminability of subtest items. Only a few items in the Noun and Verb Comprehension subtests, and one item in both Nonword Discrimination and Sentence Comprehension showed poor item discriminability (corrected item-total correlation below 0.2). The relatively lower values observed again for the Noun and Verb Comprehension subtests reflect the ceiling effects and limited spread in scores on the subtest items. Overall, the item analysis indicated a sufficient range of item difficulty across the RAT subtests and good consistency in test items. This suggests that the RAT is robustly sensitive to the full spectrum of aphasic language disorders.

Establishment of cutoff values and the RAT’s sensitivity and specificity

Cutoff values were determined differently for the subtests and for the GAQ. To calculate subtest cutoffs, we first removed outliers in the NHI group, defined as subtest scores lower than 90 percent and more than three standard deviations away from the mean value of the respective age cohort. This minimized the influence of single aberrant values on the cutoff criteria. Overall, one to two outliers were removed in several subtests in both the young and the elderly NHI cohorts. Next, for each individual subtest, the 5^th percentile of the NHI group’s score was calculated separately for the two age cohorts, which was considered the cutoff for impaired performance (see Fig 1). Performance at or below cutoff was considered abnormal, since 95% of healthy controls without a language impairment scored higher (cf., CAT [24]). The only exception to this rule was the Noun Comprehension subtest for the young cohort. In this one subtest, the 5^th percentile equaled 100%, hence the cutoff for impaired performance was adjusted to the next score possible– 95.83% (23 out of 24 trials). Otherwise, even a score of 100% would be labeled as impaired. For percentages of PWA with abnormal performance on each subtest in two age cohorts see Fig 1. It should be emphasized that while a score at or below the cutoff indicates a deficit in a given task, it does not by itself imply presence of aphasia. The latter is ascertained based on the GAQ.

To determine the diagnostic cutoff for the GAQ, a ROC-curve analysis was performed separately for each age cohort using the pROC package [41]. A ROC graph is a visual representation of a classifier where true positive rate (sensitivity, i.e. classifying a PWA as a PWA) is plotted against false positive rate (1 –specificity, i.e., classifying an NHI as a PWA) and thus demonstrates the trade-off between the two. A ROC-curve is a step function where performances of the classifier for the same dataset but at different threshold values are plotted [42]. Based on the ROC-curve, a threshold (cutoff) value can be selected that optimizes both sensitivity (positive identification of those with aphasia) and specificity (correct negative identification of those without the disorder).

In our case, two ROC-curves for two age cohorts were generated (see panel A in Fig 2). For convenience, the x-axes of the graphs were flipped to plot sensitivity against specificity. For this analysis, outliers were not removed, because the GAQ represents an average score across all subtests and is, therefore, not detrimentally impacted by aberrant values in individual subtests scores. Also, we wanted to account for a full range of possible performance in calculation of the aphasia cutoff. The optimal threshold was selected to maximize the sum of sensitivity and specificity [43]. A GAQ score at or below the cutoff indicates presence of aphasia.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Determining the cutoff score for identification of aphasia.

Panel A–Receiver Operating Characteristic (ROC) curves used to determine the General Aphasia Quotient (GAQ) cutoff score for the two age cohorts, with the x-axes flipped to plot sensitivity against specificity. The optimal threshold was selected to maximize the sum of sensitivity and specificity, as indicated by the black point on the graph. Panel B–classification accuracy for the two age cohorts according to the GAQ cutoffs.

https://doi.org/10.1371/journal.pone.0258946.g002

Two separate cutoff values for the GAQ indicating presence of aphasia were determined: 92.95% for the young cohort and 89.91% for the elderly cohort (see Fig 1), reflecting minor challenges the elderly control group experienced with some of the subtests. The RAT demonstrated excellent diagnostic accuracy: sensitivity was .938 for the young and .969 for the elderly cohort, while specificity was .985 and .96, respectively. Three and two individuals were incorrectly classified as control instead of aphasia or aphasia instead of control group in the young and elderly cohorts, respectively (see panel B in Fig 2). The missed aphasic diagnoses included two patients with very mild residual aphasia and one patient with anomic aphasia of traumatic etiology. Overall, this substantiates high sensitivity and specificity of the RAT, with its diagnostic accuracy comparable to other tests that used ROC curve analysis to determine aphasia cutoff (cf., QAB [22]).

Determination of impairment severity

We empirically defined three impairment severity ranks (mild, moderate, and severe) for each subtest and for the GAQ based on the PWA data. To find the relevant ranges, we first selected, for each subtest and for the GAQ, the PWA with abnormal performance at or below the cutoff. Then the two age cohorts were combined to increase sample size and to ensure that the calculated ranges were not influenced by our two age cohorts being unbalanced in terms of aphasia types (see Table 2 in the Methods section). Next, we divided the accuracy scores into three approximately equal ranges by calculating the 34^th and the 67^th percentiles of the subtest scores / GAQ values for these combined groups. Specifically, for both the RAT subtest scores and GAQ values the three severity ranks were defined as the following, where X_i is the individual participant’s accuracy score on a given subtest/GAQ:

• Mild: 67^th percentile ≤ X_i ≤ cutoff;
• Moderate: 34^th percentile < X_i < 67^th percentile;
• Severe: 0 ≤ X_i ≤ 34^th percentile.

Performance above cutoff was considered normal. See Fig 3 for severity ranges for subtest accuracies and the GAQ and supporting information for respective values (see S2 Table). Note that the cut-off for normal performance varies between the two groups (it is based on the 5^th percentile of the respective NHI cohort, except for Noun Comprehension–see previous section for more details), but the cut-offs for different severity ranks are identical for the two groups since we combined the two PWA age cohorts for this analysis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Severity ranks for subtest accuracy scores and the General Aphasia Quotient (GAQ) for the two PWA age cohorts.

https://doi.org/10.1371/journal.pone.0258946.g003

As can be clearly seen from Fig 3, comprehension subtests, where PWA typically score higher (except for Discourse Comprehension), had narrower moderate and mild ranges, while for repetition and production tasks these ranges were more evenly distributed. This most likely reflects several factors: Comprehension subtests being easier due to their design (multiple-choice vs. open-ended questions for repetition/production), and comprehension abilities being more spared in aphasia in general. Accordingly, as the Discourse Comprehension subtest had a safeguard built in against guessing (i.e., to receive credit, participants had to respond correctly to two questions pertaining to the same part of the story), the severity ranges were less skewed towards the higher accuracy values. In other words, individual subtests were not (and probably could not) be equated for difficulty. This makes direct comparison of raw accuracy scores between subtests uninformative, making the appraisal of severity ranks across subtests more relevant. Also, from Fig 3 it can be inferred which subtests posed the most difficulty for our aphasia sample (where lowest accuracy scores corresponded to the threshold for severe level of impairment): Discourse Comprehension, Nonword Repetition, Sentence Repetition, Object and Action Naming, and Sentence Production.

Overall, we determined these severity ranks primarily for clinical purposes. Attributing a corresponding severity level of impairment to participant’s performance on each subtest allows clinicians to qualitatively compare different subtest scores and more clearly identify spared and impaired language domains. Additionally, it allows to track the patient’s progress between different time points. However, given the intercorrelations observed between different subtest scores (see section on Construct validity below) and typical low reliability of contrast scores (e.g., [44]), with the currently available data it is not possible to reliably differentiate genuine dissociations across linguistic levels and domains from measurement errors. Therefore, between-subtest differences cannot be appraised quantitatively and should generally be interpreted with caution, particularly in individual cases.

Concurrent validity

Concurrent validity of the RAT was established by correlating the GAQ with the overall scores on the ASA [5]. A strong Pearson correlation was observed (N = 64, r = .925, p < .001, 95% CI [.879, .954]), substantiating the validity of the test (see Fig 4). However, the data also show that individuals with similar ASA total scores could obtain substantially variable GAQ scores, e.g., the same ASA score may correspond to normal, mild, or moderate impairment based on the GAQ score. On the other hand, it should also be noted that those participants identified as having mild aphasia according to the RAT demonstrated a range of scores on the ASA, but that range also fell into the interval for moderate-to-mild level of impairment according to the ASA classification (73% - 87%). While it is beyond the scope of the present paper to contrast the two tests in detail, these observations likely reflect different structures and scoring systems of the two tests. In the ASA there is greater impact of memory abilities on performance in the comprehension subtests (as participants have to comprehend strings of words of increasing length), and possibly inter-rater reliability issues with scoring of sentence and discourse production tasks (reliability of the ASA is not established). At the same time the RAT covers additional language domains (repetition), includes further language comprehension tasks (Nonword Discrimination, Lexical Decision), and affords more fine-grained assessment of language production at the sentence and discourse levels, potentially providing a more detailed evaluation of language deficits in aphasia. Accordingly, while a large correlation between the two tests supports concurrent validity of the RAT, the observed spread in scores reinforces the claim that the RAT offers additional insights and provides a more differentiative evaluation of language deficits in patients with brain damage.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Correlation between the General Aphasia Quotient (GAQ) of the RAT and the total score on the Assessment of Speech in Aphasia (ASA), a widely used aphasia battery in Russian.

Different colors indicate different severity ranks based on the GAQ.

https://doi.org/10.1371/journal.pone.0258946.g004

Construct validity

Next, we explored the inter-relationships between the RAT subtests. Notably, significant and strong Pearson correlations (Bonferroni-corrected for all pairwise comparisons) between all subtest scores were observed (see Fig 5A), as has been demonstrated previously for other language tests (QAB [22]; CAT [24]). We also performed a novel analysis that is not typically employed in aphasia batteries. We ran partial correlations (psych package for R [40]) between subtest scores accounting for aphasia severity as measured independently by the total score on the ASA [5]. Here, a more nuanced and domain specific pattern of correlations between subtests emerged after correcting for multiple comparisons (see Fig 5B). Only associations between subtests that measured similar underlying language abilities (within and across linguistic levels) continued to show a significant association, underscoring that the subtests are measuring what they intend to measure and are differentially sensitive to different underlying linguistic impairments.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Correlation between the subtests’ scores.

Panel A–simple Pearson correlations between the RAT subtests without accounting for aphasia severity. Panel B–partial Pearson correlations between the RAT subtests accounting for overall aphasia severity as measured independently by the total score on the Assessment of Speech in Aphasia (ASA). The plotted correlations are based on complete pairwise observations. The Bonferroni-corrected p-value equals .05 / 78 = 0.0006. Tiles with significant correlations are highlighted.

https://doi.org/10.1371/journal.pone.0258946.g005

Auditory comprehension subtests specifically targeting phonological and lexical levels (Nonword Discrimination and Lexical Decision) correlated only with each other and with no additional subtests. Single word comprehension subtests were also correlated. Interestingly, it was specifically verb comprehension that was found to be related to sentence and discourse level comprehension, supporting the idea of the role of the verb as a central sentence element. This correlation emerged even though most of the trials in the Sentence Comprehension subtest had syntactic but not semantic distractors. Accordingly, it is in line with the hypothesis that the verb’s grammatical properties (e.g., argument structure) are processed during comprehension and production of isolated verbs [45], linking the impairment at the single-verb and sentence levels. A similar relationship was observed between Action Naming and Discourse Production, with correlation between Action Naming and Sentence Production trending towards significance. Generally, comprehension subtests were not related to repetition and production subtests, except for the sentence-level tasks. Performance on the Sentence Comprehension subtest was related to both Sentence Repetition and Sentence Production subtests, suggesting that their performance may be relying on the same underlying processes, e.g., short-term/working memory and syntactic processing.

Repetition subtests remained highly correlated; it should be noted that specifically Nonword Repetition correlated strongly with Sentence Repetition likely indicating critical involvement of the phonological buffer in both tasks. As expected, Object and Action Naming subtests were related. Both naming subtests were also related to Nonword and Word Repetition subtests, but again it was specifically Action Naming that was related to the Sentence Repetition subtest and the Discourse Production subtest. Overall, the prominent relationships observed between performance on subtests targeting verb processing and sentence/discourse level tasks highlights the importance of assessing verb comprehension and action naming in aphasia. Future studies are needed to explore these differential association patterns in greater detail.

Internal reliability

We estimated internal reliability of subtest scores using Cronbach’s alpha [46] calculated with the psych package for R [40]. While the Cronbach’s alpha has limited application for cases when the underlying construct is multi-dimensional [47], as is the case with most language subtests, it can provide at least an approximate lower-bound estimate of reliability even when tau-equivalence (i.e., the same true score for all test items, or equal factor loadings of all items in a factorial model) is violated. Internal reliability of RAT subtest scores was good (Noun and Verb Comprehension, Sentence Comprehension) to excellent (Nonword Discrimination, Lexical Decision, all repetition and production subtests), based on the criteria outlined by Koo and Li [48]. Thus, this provides preliminary evidence on the consistency of the items comprising the different subtests.

Inter-rater reliability

Inter-rater reliability for repetition and production subtests was calculated based on 20 evaluations of PWA responses (the inter-rater PWA group, see Table 1 in the Methods section). Each subject’s response was evaluated independently by two trained raters according to the detailed scoring instructions. Inter-rater reliability was not computed for the comprehension subtests since those subtests are scored automatically by the tablet-based application. The intraclass correlation coefficients, absolute agreement (ICC, type A-1) for each subtest were calculated using package irr [49] and are shown in Table 4. The ICCs ranged from .833 (Discourse Production) to .994 (Object Naming). These ICC values indicate excellent inter-rater reliability for all subtests except Discourse Production, according to the criteria defined by Koo and Li [48]. The relatively lower (although still good) reliability of the Discourse Production subtest is a consequence of the more subjective nature of the scoring system for this task and is comparable to (or even exceeds) the inter-rater reliability values reported for other discourse rating scales [50]. No significant differences were observed between ratings provided by the two raters (paired-sample t-tests, p > .05). Overall, these results demonstrate that with the provided scoring instructions, the RAT can be reliably scored by trained clinicians.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Reliability estimates of the RAT subtests: internal reliability estimated with Cronbach’s alpha, inter-rater and test-retest reliability based on the intraclass correlation coefficients, absolute agreement (ICC, type A-1).

https://doi.org/10.1371/journal.pone.0258946.t004

Test-retest reliability

Test-retest reliability was calculated based on 20 additional individuals with chronic aphasia who were evaluated two times each on two separate occasions (the test-retest PWA groups, see Table 1 in the Methods section), with no speech-language therapy in between. The ICCs (type A-1) for test-retest reliability for each of the subtests (calculated again using package irr [49]) are presented in Table 4. All repetition and most oral production subtests demonstrated excellent test-retest reliability according to the stringent criteria outlined by Koo and Li [48], with Verb Naming showing good and Discourse Production–moderate reliability. The lower reliability of the Discourse Production subtest, again, likely reflects the more subjective nature of the scoring for this subtest (as discussed in the previous section). Comprehension subtests showed variable reliability: more complex subtests revealed good (Lexical Decision, Sentence Comprehension, Discourse Comprehension) and moderate (Nonword Discrimination) reliability, while the single word comprehension subtests demonstrated poor reliability (Noun and Verb Comprehension). These low ICCs reflect high probability of responding at chance (i.e., limitations of a forced choice format), limited variance, and the ceiling effects observed in these subtests (cf., CAT [24], QAB [22]). If compared directly, the differences in scores between the two testing sessions for any of the RAT subtests were trivial and non-significant (paired-sample t-tests, p > .05). This observation in combination with good to excellent internal reliability estimates for all comprehension subtests implies that the below-normal scores on these subtests can still be confidently interpreted as such. Still, given that particularly single word comprehension subtests demonstrated poor test-reliability reliability, we would caution future test users against interpreting them in isolation or in contrast scores, especially for individual cases.

Also, since test-retest data was collected across two groups of PWA, we were not able to compute test-retest reliability of the GAQ. However, to get an estimate of the reliability of the overall test score, we computed a partial GAQ based on the available subtest scores in each of the two groups. ICCs of the GAQ-proxy were high in both instances supporting projected excellent reliability of the summary score. Taken together, the results of internal, inter-rater, and test-retest reliability analyses demonstrate that the RAT scores are highly stable across test administrations, with possible minor fluctuations in comprehension scores.

Limitations

While the development and the introduction of the RAT into clinical practice presents a great advancement in comprehensive language assessment of Russian-speaking people with aphasia, the collected dataset has several limitations, which will need to be addressed by future investigations.

First of all, since technical errors precluded collection of test-retest data for all of the RAT subtests in a single sample, we could not establish test-retest reliability values for the overall test score–the GAQ. Since test-retest reliability of most of the RAT subtests is high along with high reliability of partial GAQ scores, we anticipate that overall test reliability will be excellent as well, however this still needs to be empirically demonstrated.

Next, more work is required to characterize the internal structure of the RAT. A much larger sample of PWA is needed to properly identify the test’s underlying factors and evaluate the internal structure of the subtests using specific estimators [47, 51].

Furthermore, we did not directly investigate sensitivity of the test to detecting a clinically meaningful change. Future studies that document behavioral changes with criterion-based tasks could inform the sensitivity of the test to detecting improvement (or decline) in language functioning and provide guidance to clinicians on how to interpret observed changes in the scores in a meaningful way. Regression-based approaches for determining clinically significant changes in the scores should also be explored in future work (e.g., [52]).

Another limitation of the current normative dataset is that we only collected data from individuals with brain damage who were diagnosed with aphasia. Future inquiries need to include persons with brain damage but without language impairments, so that sensitivity of the test specifically to language deficits rather than to general cognitive sequalae of brain injury (e.g., fatigue, compromised attention, memory, executive skills) is ascertained. While the selective relationships observed between different RAT subtests demonstrate that subtests are differentially sensitive to specific language deficits and underscore the test’s construct validity, data from a non-aphasic brain injury group would further raise the diagnostic value of the RAT.

Finally, the RAT currently does not assess reading and writing abilities. We hope that in the near future we will be able to develop and standardize these subtests as well.