Article Text
PDF
Abstract
Introduction Early detection and tracking of bulbar dysfunction in amyotrophic lateral sclerosis (ALS) are critical for directing management of the disease. Current clinical bulbar assessment tools are lacking, while existing physiological instrumental assessments are often inaccessible and cost-prohibitive for clinical application. The goal of our research is to develop and validate a brief and reliable, clinician-administered assessment tool—the ALS-Bulbar Dysfunction Index (ALS-BDI). This publication describes the study protocol that has been established to ascertain the tools’ psychometric properties.
Methods and analysis The ALD-BDI’s development closely follows guidelines outlined by the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN). Through the proposed study protocol, we expect to establish psychometric properties of both individual test items of the ALS-BDI as well as the final version of the entire tool, including test–retest and inter-rater reliability, construct validity using gold-standard assessment methods and responsiveness.
Ethics and dissemination This study has been reviewed and approved by research ethics boards at two data collection sites: Sunnybrook Health Science Centre, primary (Toronto, Canada; ID3080) and Mass General Brigham (#2013P001746, Boston, USA). Prior to participation in the study, the participants sign the informed consent in accordance with the Declaration of Helsinki. Once validated, the ALS-BDI will be disseminated to key stakeholders. Following validation, the ALS-BDI and any required training material will be implemented for clinical use in a context of a multidisciplinary ALS clinic and used as an outcome measure for clinical trials in ALS research.
- motor neurone disease
- rehabilitation medicine
- neuromuscular disease
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
The Amyotrophic Lateral Sclerosis - Bulbar Dysfunction Index (ALS-BDI) development process adheres to the established methodological guidelines for developing a formative assessment measure—COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN).
Initial development and design of the ALS-BDI has involved extensive consultation of projected end-users (ie, neurologists, speech–language pathologists) to ensure that it is comprehensive, clinically feasible and aligned with current practice goals and workflow.
Psychometric evaluation of the ALS-BDI will include test–retest and inter-rater reliability, construct validity and responsiveness of each test item and the overall tool.
Patient anchors will be used to inform interpretability and determine the clinical significance of ALS-BDI scores, including the minimal important difference and low activity disease states.
Many individual items on the drafted ALS-BDI involve a subjective rating of severity (ie, mild, moderate, severe), which may challenge inter-rater reliability.
Introduction
Amyotrophic lateral sclerosis (ALS) is a fast-progressing disease with a lifespan post-diagnosis of 2–5 years.1 The disease is characterised by the degeneration of motor neurons in the brain, brainstem and spinal cord as well as extramotor (eg, cognitive–linguistic) brain pathways.2–4 The degeneration of motor neurons results in progressive muscle weakness, atrophy and eventual paralysis. Nearly 90% of individuals diagnosed with ALS will experience speech and swallowing dysfunction either at the onset of the disease or when the disease spreads to the bulbar motor system, which controls muscles of the head and neck.5 The presence of bulbar motor impairment is associated with a shorter survival (<2 years) and increases the risk of death by almost eightfold.6 7 The resulting speech impairment has been rated as the worse aspect of the disease by patients with ALS.8 Despite the devastating consequences of bulbar dysfunction on the survival and quality of life, there are currently no validated tools for its assessment in a clinic. In the absence of a standardised assessment of bulbar motor dysfunction, current practices in ALS clinics are idiosyncratic, piecemeal and rely primarily on symptom checklists.9
The overall aim of our work is to develop a reliable and psychometrically validated clinician-administered assessment tool of bulbar dysfunction, called the ALS-Bulbar Dysfunction Index (ALS-BDI), that is efficient, standardised, clinically feasible, comprehensive and responsive to change over time. The ALS-BDI is currently being designed and validated for two diagnostic use cases: (1) to discriminate among patients with different severities of bulbar impairment (ie, discriminative purpose); and (2) to evaluate the changes over time in a person or group (ie, evaluative purpose).10 This tool will have immediate implications for tracking disease progression and monitoring changes in ALS clinics or in clinical trials. In the future, it may serve as the core assessment that can be adapted for improving ALS diagnosis and prognosis; it could also eventually serve as a screening tool for determining the need for full dysphagia and speech assistive technology evaluations.
Our process for developing the ALS-BDI adheres to established guidelines for developing a formative measure (COnsensus-based Standards for the selection of health Measurement INstruments, COSMIN).11 12 Within a formative model, all items are considered relevant to represent the overall construct of interest and are not interchangeable; items may or not be correlated, and they may have different patterns of change when the construct changes. Formative measures, as opposed to reflective measures, are not assessed with respect to internal consistency or item-total correlations such as Cronbach’s alpha, factor analysis or the item response theory. The decision to retain or reduce items is primarily based on their clinical relevance in defining the construct (content and face validity), although reliability and responsiveness can also inform item retention.
Our test development process is divided into three development cycles: Cycle 1 includes generation of the candidate item pool and demonstration of their content and face validity; Cycle 2 is focused on establishing the item and overall tool’s reliability; and Cycle 3 establishes the item and overall tool’s construct validity in relation to well established instrumental and clinical measures, as well as responsiveness to change over time. The work in Cycle 1, which produced the first iteration of the ALS-BDI, has been now completed. The initial development steps have been described in detail elsewhere.13 Briefly, Cycle 1 involved convening a panel of experts to identify key assessment domains for bulbar dysfunction. The panel met during a Northeastern ALS Consortium (NEALS) Bulbar Subcommittee meeting, members of which were experts in neurology, speech–language pathology (SLP) and measurement science.14 Through consensus, the panel identified three core bulbar assessment domains: (1) Cranial Nerve Exam; (2) Auditory-Perceptual Assessment (of speech and voice); and (3) Functional Assessment (of speech intelligibility, swallowing, chewing and coughing). Following the meeting, a literature review was conducted by our development team to generate candidate items for each domain. Expert surveys were then conducted to probe each item’s face and content validity. Based on this procedure, an initial draft of ALS-BDI was created. A group of SLPs further vetted the testable draft of the instrument via pilot administration of the test draft and completion of a cognitive interview with a member of our development team. As a result, a beta of ALS-BDI has been prepared for its psychometric evaluation.
The process for establishing psychometric properties of the ALS-BDI has been outlined in the recommendations by the COSMIN group.11 To our knowledge, no comprehensive bulbar assessment tool has yet been developed and validated using this rigorous approach.15 The key psychometric properties of the ALS-BDI that must be assessed under this framework include inter-rater and test–retest reliability (Cycle 2 of the development), as well as the construct validity against gold-standard measures of bulbar dysfunction in ALS and responsiveness to change and minimal important difference (MID), minimal detectable change (MDC) and low disease state of ALS-BDI to enhance interpretability of its scores (Cycle 3 of the development). Here we present a detailed protocol for the psychometric evaluation of ALS-BDI through Cycle 2 and Cycle 3.
Methods and analysis
Prospective study design
A prospective longitudinal study design will be used to assess the ALS-BDI.
Participants and recruitment
Data for this study will be collected at the Bulbar Function Laboratory at the Sunnybrook Health Sciences Centre in Toronto, Ontario, Canada, and the Speech and Feeding Disorders Lab at the MGH Institute for Health Professions in Boston, Massachusetts, USA. The study has been initiated in May 2021 with anticipated completion by June 2023.
Possible, probable or definite ALS was diagnosed in 100 adults (≥18 years) as defined by the Revised El Escorial Criteria will be recruited consecutively at the ALS/MND Clinics in Toronto, Canada and Boston, USA.16 Individuals will be excluded if they have a history of other neurological conditions (eg, stroke) or head and neck cancer; take medications known to affect speech production; are unable to comply with the study protocol due to significant cognitive impairment (ie, Montreal Cognitive Assessment score <18);17 or fail a hearing screen in both ears (ie, 500–4000 Hz, thresholds >40 dB).
To ensure that the ALS-BDI is tested across a broad sampling of disease severity, we will recruit an equal number of patients who fall within four categories of bulbar symptom severity, based on their ALS Funtional Rating Scale - Revised (ALSFRS-R) bulbar subscore: normal bulbar function (ie, score of 12/12); mild (9–11/12); moderate (6–8/12); or severe (1–5/12).18
Inter-rater and test–retest reliability
To estimate the inter-rater reliability of each candidate item, two SLPs will rate each patient’s performance on the ALS-BDI on the same visit. The order of SLP raters will be randomised, and they will be blinded to each other’s ratings. The first administration will be conducted at the beginning of the visit, and the second will take place at the very end. To minimise effects of fatigue, participants will be given sufficient time to rest before the second ALS-BDI assessment.
To determine test–retest reliability, each participant will be reassessed using the ALS-BDI, within 1 week after the initial visit. The second visit will be scheduled approximately at the same time of day as the first visit, to mitigate any daytime variability in fatigue. Given the natural history of ALS, it is unlikely that there will be significant clinical change in the period of 2 weeks. However, on the second visit, the participants will answer whether they think their bulbar function has changed compared with the first assessment (ie, yes/no). Participants who report any perceived differences in bulbar symptoms compared with the initial visit will be excluded from the test–retest reliability calculations.
We will calculate reliability statistics for each item on the ALS-BDI using weighted Kappa, because individual items were scored ordinally. Items with weighted Kappa <0.6 on either inter-rater or test–retest reliability will be flagged for reduction.19 We will also calculate the overall reliability of the total score of the ALS-BDI (ie, the sum of all items), using intra-class correlation coefficients (ICCs), considering raters as random effects (ICC formula 2,1).20 21 ICCs of ≥0.80 will be set as the minimal acceptable standard for evaluating groups of patients, whereas values of ≥0.90 will be required for individual patient use.22 Item reduction based on reliability will be completed prior to construct validation of individual items and the full ALS-BDI.
Data from the first 50 consecutive individuals diagnosed with ALS and presenting with a range of bulbar impairment will be used for the reliability analyses. This exact sample size was estimated using the approach by Kraemer and Korner,23 whereby for a minimum ICC of 0.8 and a lower 95% CI of 0.7, 47 analysable participants should be required. We will increase this number if some patients fail to attend the second visit (test–retest) within 2 weeks of the initial visit recording, or if they report a change in clinical status during this time interval.
Construct validity
Items remaining following reliability assessment will be evaluated for construct validity, relative to gold-standard instrumental physiological measures of bulbar dysfunction. For this phase of the study, data from 100 participants with ALS will be recorded. On the day of ALS-BDI administration, each patient will complete a battery of instrumental assessments and selected patient-reported outcomes indicating overall ALS and bulbar-related disability, namely the ALSFRS-R and the Center for Neurologic Study Bulbar Function Scale (CNS-BFS).18 24
Table 1 shows the detailed mapping between individual items of ALS-BDI and specific instrumental validators (ie, tasks and measures). The entire session will be audio-recorded at high resolution (44.1 kHz, 16-bit) using a unidirectional lapel microphone, with a fixed mouth-to-mic distance; the audio signal will be calibrated using a 1000 Hz tone at the start of each session to ensure accurate measures of signal intensity.25 In addition to audio recordings of all speech tasks, the Phonatory Aerodynamic System (Pentax Medical) will be used to measure nasal airflow and oral pressure during speech and cough.25 26 Tongue and lip/facial muscle strength will be measured using the Iowa Oral Performance Instrument (IOPI Medical LLC) during maximum pressure generation tasks.27 28 Facial movements for speech and non-speech (oromotor) will be recorded using a three-dimensional video camera (Intel RealSense),29 and tongue movement will be assessed using the Wave Speech System (NDI).30 31 To validate swallow tasks, each participant will undergo a standard videofluoroscopic swallow study. The full validation protocol is estimated to take approximately 2 hours.
Validation procedures for ALS-BDI items
All speech tasks will be analysed using well-established spectral and timing measures (eg, voice onset time, acoustic vowel space, segment durations; see table 1 for examples).25 The standard passage reading will be analysed using an automated Speech-Pause Analysis algorithm, extracting measures of phrase and pause durations.32 In addition, participants will complete the Speech Intelligibility Test to quantify speech intelligibility (ie, % words heard correctly) and speaking rate (ie, words per minute) in sentences.33 Measures of swallow safety (ie, penetration/aspiration),34 efficiency (ie, residue)35 and key physiological events (eg, pharyngeal constriction,36 37 chewing duration, laryngeal vestibule closure)35 will be obtained from a videofluoroscopic swallow study. Cough volume acceleration will be measured from the Phonatory Aerodynamic System waveform of voluntary cough.38 39
We will study the validity of each reliable item, as determined in the reliability assessment described above, in relation to its corresponding instrumental measure. All instrumental measures for this study are continuous, so we will calculate the correlations (Pearson’s or Spearman’s as appropriate) between each item and its instrumental equivalent. We have set an a priori correlation cut point of ≥0.6 for criterion validity; items that correlate with instrumental measures of >0.6 will be flagged for inclusion.40 Inter-item correlations will also be examined for redundancy. In a formative model, while low or absent correlations between items are not an issue, very high inter-item correlations (>0.9) can indicate redundancy; pairs of items with correlations >0.9 will be assessed for potential reduction. If additional items are eliminated, we will re-calculate the reliability coefficients for the final version of the ALS-BDI.
Once we have the final version of the ALS-BDI, we will study the construct validity of the entire tool. We will test two hypotheses for construct validity: (1) the ALS-BDI total score will be highly correlated (r≥0.7) with ALSFRS-R bulbar subscore and the CNS-BFS, which measures bulbar-specific disability; (2) the correlation will be weaker (r=0.4–0.6) with the ALSFRS-R total score, which measures ALS disability as a whole and includes the effect of limb weakness and respiratory dysfunction. We will also compare the ALS-BDI scores between subgroups of patients divided by bulbar function severity. We will classify patients as having normal, mild, moderate and severe bulbar dysfunction, based on their ALSFRS-R bulbar subscores.18 We will compare the mean ALS-BDI scores on each bulbar severity group using analysis of variances. We hypothesise that the different severity groups will have statistically significant different mean ALS-BDI scores. Following the COSMIN guidelines, we will consider that the new scale has construct validity, if we confirm ≥75% of the predefined hypotheses.41
Establishing responsiveness, MID, MDC and low disease state of ALS-BDI
For this analysis, patients will have a second full assessment 6 months after the initial visit. On the second visit, patients will undergo the ALS-BDI, instrumental testing (as described above), as well as the ALSFRS-R and CNS-BFS. Patients will also answer a patient impression-of-change questionnaire (PIC), to indicate if their speech, voice and/or swallowing have changed since the first assessment. The PIC will have four possible answers: 0=no change, 1=minimally worse, 2=much worse, 3=very much worse; this is a modification from the method by Juniper, that used seven options for improvement or worsening.42 Considering the natural history of ALS, we do not expect patients to report improvement. We will calculate the difference in ALS-BDI scores between visits 1 and 2, as well as the change scores for the instrumental measures, the ALSFRS-R and CNS-BFS. There is no single way to determine responsiveness, so current guidelines recommend using a range of methods to document sensitivity to change including (1) assessing statistically significant change—such as in a clinical trial—by comparing mean effects with a t-test, and (2) assessing change that is meaningful for patients.41
We will estimate statistically significant change in four ways. First, we will compare the mean ALS-BDI scores at session 1 and session 2 through paired t-tests; we expect to see a statistically significant difference (p<0.05), indicating worsening of bulbar function. Second, we will calculate the magnitude of change through the standardised response mean (SRM), which is the ratio between the mean change score to the SD of said change score. We hypothesise that the SRM will be ≥0.3. Third, we will study the correlations between the change in the ALS-BDI and the comparison measures, as a form of longitudinal validity.43 We expect to find moderate correlations between the change scores (r=0.4–0.7). Fourth, we will study the efficiency to detect statistically significant change of the ALS-BDI compared with the instrumental measures, the ALSFRS-R bulbar score and CNS-BFS. For this, we will calculate the ratio of the paired t-test statistics, as follows:
.44 45
A ratio >1 indicates that the ALS-BDI is more efficient than the comparison measure of interest; in other words, it is more sensitive with detecting a given effect size for the same sample size. We hypothesise that the ALS-BDI will be more efficient than the ALSFRS-R bulbar score and the CNS-BFS; we expect that the ALS-BDI subscores will have equal or slightly lower efficiency than the corresponding instrumental measures.
We will determine if the ALS-BDI detects change that is meaningful to patients by using the PIC as an anchor. To be meaningful from the patients’ perspective, the ALS-BDI change-scores should correlate with the PIC. For instance, patients reporting to be much worse should have a higher change in score than those reporting moderate, mild or no change. We will use analysis of covariance to compare the mean change in scores among PIC groups, correcting for differences in baseline scores. We expect a significant difference in change scores across PIC groups, with larger change scores in those patients reporting to be much worse.
To enhance the interpretation of the change in scores, we will estimate the MID for worsening. This is the smallest change in score that is meaningful for patients, and we will use the PIC category of ‘a little worse’ as the anchor of minimal worsening. This follows current recommendations of using anchor-based methods to determine the MID, as opposed to distribution-based methods.46 We will estimate the MID at the individual level—that is, to classify one individual as having at least minimal worsening—using a receiver operator characteristic curve. Patients who have at least minimal worsening will be classified as ‘changers.’ The MID at the individual level is the point of highest sensitivity and specificity. We will also estimate the MID at the group level—that is, the smallest mean change in scores that would be meaningful in a group, such as in a clinical trial. For this, we will calculate the mean change in ALS-BDI scores in the group of patients that reported being ‘a little worse’42; we will also estimate the 75% percentile for potential misclassification bias. For the MID estimates to be interpretable, they should be above error of measurement. Therefore, we will calculate the MDC, which is the smallest change between assessments that is very likely above error of measurement. The MDC (with 95% CI) is calculated using the test–retest reliability statistic as follows:
where SD is the SD of the sample, and ICC is the test–retest reliability coefficient.46 For the MID scores to be meaningful, they should be above error; the MID should be larger than the MDC.
We will use the patient anchor to determine an ALS-BDI score threshold for the low disease activity states.47 This metric is similar to the patient acceptable symptom states (PASS), which reflect the threshold on a patient-reported outcome where people feel generally well. At the same time, it differs from the MID, described above, as it is not a measure of change. To determine this threshold, we will ask patients on both visits, ‘Considering all the difficulties on speaking and swallowing that you experience due to ALS, if you were to remain at this same level of function for the following months, would you consider your speaking or swallowing function satisfactory?’. This is a modification of PASS/low disease state questions that have been used in other diseases to help with interpretability of the scale scores.47–50
Patient and public involvement
We envision the end users of ALS-BDI to be clinicians delivering care to patients with ALS in the context of a multidisciplinary ALS clinic. As such, patients have not been directly involved in the development of the tool but were engaged in the trial assessments with the tool in Cycle 1 of its development. Patients have not been directly involved in the design of the reliability or validation studies presented here (Cycles 2 and 3). Patients were involved in the pilot of the validation study to ensure that the study is well tolerated and optimised for the abilities of those patients with a more advanced presentation of ALS. Patients will be involved in conduct of the study. The results will be disseminated to the patient participants through sharing study publications and poster presentations with those who will express interest in receiving this information. The results will also be presented to patient groups.
Discussion
Bulbar motor dysfunction is among the most devastating consequences of ALS, a fast-progressing motor neuron disease. Although the assessment of bulbar dysfunction is a core component of the ALS clinical examination, a well-designed, easy to perform and efficient tool for its assessment is lacking.14 15 This work aims to fill this gap by developing and validating a clinician-administered tool, the ALS-BDI, to serve as a primary bulbar assessment tool in a clinic and in clinical trials. The innovation of this work is not only in its uniqueness and methodological rigour, but the state-of-the-art instrumentation that will be used for its validation.
The ALS-BDI is being designed for SLPs because of their specialised training in the assessment of voice, speech and swallowing impairments. The tool’s administration is consistent with clinical assessment procedures used by SLPs, which are based on clinician ratings of speech, swallowing and oral structure and function. Although clinician ratings are the foundation of an SLP assessment, the efficacy of these measures has rarely been tested using rigorous experimental designs.51
Because the tool will be based on the clinician’s ratings, particular attention will be paid to creating a training module for its standardised administration. Further, the tool’s efficiency will be addressed by careful evaluation and selection of the items. A useful bulbar assessment will need to be briefed, such that it can be administered from start to finish within 10–15 min, even for patients who are severely affected. A brief assessment will reduce the burden on the patients, who are prone to fatigue and are frequently evaluated by multiple practitioners within a single clinic visit. Standardisation and efficiency will be achieved by developing a tool with a minimal set of test items—removed based on reliability, validity and responsiveness evaluations—that selectively target key clinical characteristics associated with bulbar ALS and is accompanied by explicit instructions for administration.
Next steps and impact
The proposed study was placed on hold between March 2020 and March 2021 due to the global COVID-19 pandemic. During this time, the tool was redesigned for an online/remote administration, and the summary of the remote version of the tool is forthcoming. Meanwhile, the return to in-person study administration outlined in the current protocol is in progress.
To the best of our knowledge, the ALS-BDI will be the first standardised and psychometrically validated clinician-administered bulbar assessment tool. The ALS-BDI aims to meet the key requirements for the development of new clinical assessment tools: (1) supported by the multidisciplinary ALS expert consensus; (2) developed with considerations of efficiency, specificity and standardisation; and (3) established evidence of strong psychometric properties. The long-term impacts of the ALS-BDI includes improving detection of bulbar ALS, expediting diagnosis, improving clinical decision-making and accelerating ALS clinical trials and drug discovery. Because the tool is consistent with current assessment practices in SLP, which are based on clinician ratings of speech, swallowing and oral structure and function, it is likely to have uptake within clinical practice settings focused on ALS.
Ethics and dissemination
This study has been reviewed and approved by research ethics boards at both data collection sites: Sunnybrook Health Science Centre, primary (Toronto, Canada; ID3080) and Mass General Brigham (Boston, USA; #2013P001746). Prior to participation in all elements of the study, the participants sign the informed consent in accordance with the Declaration of Helsinki. Once validated, the ALS-BDI will be disseminated to key stakeholders at clinical ALS and SLP conferences (eg, American Speech Hearing Association (ASHA) Convention, International Symposium on ALS/MND). The tool and any relevant training materials will be made publicly available to SLPs and neurologists who provide ALS care. The ALS-BDI will also be launched as a validated outcome tool for clinical research trials in ALS.
Ethics statements
Patient consent for publication
Acknowledgments
We would like to thank all members of the Northeastern ALS Consortium Bulbar Subcommittee, who participated in focus groups, as well as the speech–language pathologists and neurologists who participated in online surveys and cognitive interviews, for providing important input into the domain conceptualisation and informing design and drafting of the Amyotrophic Lateral Sclerosis – Bulbar Dysfunction Index. We also extend appreciation to Madhura Kulkarni and Brian Richburg for managing the study, Diego Guarin and Andrea Bandini for designing key tools for the validation protocol and Amanda Martino and Reeman Marzouqah for pilot testing the validation protocol.
References
Footnotes
Twitter @YunusovaLab
Contributors YY, CB, RM, AA, LZ, JDB, GLP and JRG conceived the study and prepared the funding application. All authors contributed to the design of the study protocol. AH and AW coordinated pilot testing of the validation protocol. All authors reviewed feedback and came to consensus on tool design. YY, AW, CB and JRG wrote the manuscript. All authors read, edited and approved the final manuscript.
Funding This work has been supported by National Institute on Deafness and Other Communication Disorders (NIDCD) grant number R01DC017291.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; peer reviewed for ethical and funding approval prior to submission.