Improving Measurement Methods in Rehabilitation: Core Concepts and Recommendations for Scale Development

doi:10.1016/j.apmr.2012.06.001

Archives of Physical Medicine and Rehabilitation

Volume 93, Issue 8, Supplement, August 2012, Pages S154-S163

https://doi.org/10.1016/j.apmr.2012.06.001 Get rights and content

Abstract

Velozo CA, Seel RT, Magasi S, Heinemann AW, Romero S. Improving measurement methods in rehabilitation: core concepts and recommendations for scale development.

Validated measurement scales are essential to evaluating clinical outcomes and conducting meaningful and reliable research. The purpose of this article is to present the clinician and researcher with a contemporary 8-stage framework for measurement scale development based on a mixed-methods qualitative and quantitative approach. Core concepts related to item response theory are presented. Qualitative methods are described to conceptualize scale constructs; obtain patient, family, and other stakeholder perspectives; and develop item pools. Item response theory statistical methodologies are presented, including approaches for testing the assumptions of unidimensionality, local independence, monotonicity, and indices of model fit. Lastly, challenges faced by scale developers in implementing these methodologies are discussed. While rehabilitation research has recently started to apply mixed-methods qualitative and quantitative methodologies to scale development, these approaches show considerable promise in advancing rehabilitation measurement.

Section snippets

Core IRT Concepts

IRT is a paradigmatic and methodological approach for the development, psychometric analysis, and scoring of measurement scales. In contrast to classical test theory, which focuses on the entire test, IRT focuses on the item (ie, scale question) and the relationship of item responses to a single underlying ability or latent trait.¹⁵ A core principle of IRT is that the probability of correct responses to an item is a function of “person ability” and the parameters of the item, notably item

Qualitative Methods

Scale developers build item pools that must meet the IRT assumption of unidimensionality, span the continuum of the latent trait from very easy to very difficult, and represent the real-life experiences of the population to be measured. We recommend the use of structured qualitative methods to conceptualize the construct(s) to be measured and to develop the items that represent the construct. Qualitative methods used in the first 3 stages of measurement scale development include context

Qualitative Approaches

While the development of a mixed-methods approach to rehabilitation measurement is timely, it faces a number of challenges. Three of these challenges include (1) replication, (2) compatibility across methodologies, and (3) the demonstration of psychometric benefit. While the systematization and rigor of qualitative methodology continue to improve, questions regarding the replication of results persist. For example, would a Delphi process using 2 different sets of experts produce similar

Summary and Conclusions

Rehabilitation research has been at the forefront of the use of mixed-methods approaches and IRT applications. Despite independent advances in qualitative and quantitative research methods, the integration of these methodologies is fairly new to scale development in rehabilitation and health care. A deeper understanding of these methodological techniques is needed to provide the most informative measures for evaluating patients while maximizing efficiency and precision. Despite these

References (63)

J.M. Linacre et al.
The structure and stability of the Functional Independence Measure
Arch Phys Med Rehabil
(1994)
S.M. Haley et al.
Pediatric Evaluation of Disability Inventory: clinical interpretation of summary scores using Rasch rating scale methodology
Phys Med Rehabil Clin N Am
(1993)
S.R. Rosas et al.
The use of concept mapping for scale development and validation in evaluation
Eval Program Plann
(2007)
D.L. Patrick et al.
Content validity—establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2—assessing respondent understanding
Value Health
(2011)
M.L. Woodbury et al.
Dimensionality and construct validity of the Fugl-Meyer Assessment of the upper extremity
Arch Phys Med Rehabil
(2007)
L.M. Avery et al.
Rasch analysis of the Gross Motor Function Measure: validating the assumptions of the Rasch model to create an interval-level measure
Arch Phys Med Rehabil
(2003)
M. Rose et al.
Evaluation of a preliminary physical function item bank supported the expected advantages of the Patient-Reported Outcomes Measurement Information System (PROMIS)
J Clin Epidemiol
(2008)
D.L. Hart et al.
Computerized adaptive test for patients with knee impairments produced valid and responsive measures of function
J Clin Epidemiol
(2008)
S.M. Haley et al.
Measurement precision and efficiency of multidimensional computer adaptive testing of physical functioning using the Pediatric Evaluation of Disability Inventory
Arch Phys Med Rehabil
(2006)
U.S. Department of Health and Human Services. Research plan for the National Center for Medical Rehabilitation...

Disability and rehabilitation research projects and centers program—Rehabilitation Research and Training Centers (RRTCs)—employment policy and measurement

Federal Register

(2010)

B. Silverstein et al.

Applying psychometric criteria to functional assessment in medical rehabilitation: IIDefining interval measures

Arch Phys Med Rehabil

(1992)

D. Cella et al.

The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years

Med Care

(2007)

AHRQ Patient Centered Outcomes Research Institutional Award (K12) 2011

PCORI funding announcement pilot projects grants 2011

Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims

G. Frank et al.

Qualitative research in occupational therapy: from the first to the second generation

OTJR (Thorofare N J)

(2010)

A. Ohman

Qualitative methodology for rehabilitation research

J Rehabil Med

(2005)

K.E. Lasch et al.

PRO development: rigorous qualitative research as the crucial foundation

Qual Life Res

(2010)

A. Onwuegbuzie et al.

Mixed research as a tool for developing quantitative instruments

J Mix Methods Res

(2010)

D.A. DeWalt et al.

Evaluation of item candidates: the PROMIS qualitative item review

Med Care

(2007)

B.B. Reeve et al.

Applying item response theory modeling for evaluating questionnaire item and scale propertiesAssessing quality of life in clinical trials: methods of practice

B.D. Wright et al.

Observations are always ordinal; measurements, however, must be interval

Arch Phys Med Rehabil

(1989)

B.D. Wright et al.

Rasch and Wright: the early years

Rasch Meas Trans

(1987)

F. Samejima

Estimation of latent ability using a response pattern of graded scores

Psychometrika Monogr

(1969)

T.G. Bond et al.

Applying the Rasch model: fundamental measurement in the human sciences

(2007)

W.J. van der Linden et al.

Handbook of modern item response theory

(1997)

R.K. Hambleton

Principles and selected applications of item response theory

D. Andrich

Controversy and the Rasch model: a characteristic of incompatible paradigms?

Med Care

(2004)

H. Wainer et al.

Computerized adaptive testing: a primer

Qual Life Res

(2001)

M.K. Trochim et al.

Concept mapping for planning and evaluation

(2007)

Cited by (45)

Measurement Properties of the MPAI-4 in Veterans With mTBI
2020, Archives of Physical Medicine and Rehabilitation
Citation Excerpt :
In step 1 of the data analysis (see fig 1), items were removed if they showed high residual correlation with other items (>0.4 considered to violate the assumption of local independence). Rasch analysis is based on item response theory, which focuses on the association between items in a questionnaire and the latent trait that is being measured.34 Item difficulty and person ability are calibrated and aligned along a linear continuum or hierarchy.
To determine whether the measurement properties of the Mayo-Portland Adaptability Inventory Version 4 (MPAI-4) and its participation index (M2PI), which have been adopted as 2 outcome measures in the Veterans Health Administration (VHA) National Polytrauma Rehabilitation Systems of Care, are adequate in veterans with mild traumatic brain injury (mTBI).
Cross-sectional.
Outpatient rehabilitation.
Postdeployment veterans with blast-related mTBI (N=177).
Not applicable.
Mayo-Portland Adaptability Inventory Version 4 (MPAI-4) and Community Integration Questionnaire (CIQ).
The unidimensional factor structure of the MPAI-4 total and 3 index scores (abilities, adjustment, participation) were confirmed. Eight of the 30 items were removed for violating monotonicity (6 items) and exceeding Rasch infit (2 items). The rating scale was collapsed from 5 to 3 ratings because of structure issues. The remaining 22 MPAI-4 items demonstrated excellent item/person reliability (0.98/0.91) and separated person ability into 4 strata. Two of the MPAI-4 index scores (abilities and adjustment) had good measurement properties. The third index, M2PI, retained only 3 items that had adequate person reliability (0.75) but separated person ability into only 2 strata. A significant but fair association with the CIQ was demonstrated with the modified MPAI-4.
The MPAI-4 has been validated in moderate to severe traumatic brain injury but required modification when used in active military personnel with mTBI. We also identified the need for modification of the MPAI-4 to support adequate psychometrics when measuring outcomes in veterans with mTBI. Additional validation of the M2PI is needed in veterans and active military personnel with mTBI, to determine whether the M2PI should continue to be used as an outcome measure in the VHA polytrauma rehabilitation systems.
Validation of the Japanese version of the life satisfaction checklist (LiSat-11) in patients with low back pain: A cross-sectional study
2018, Journal of Orthopaedic Science
Previous studies suggest that life satisfaction assessment using the Life Satisfaction checklist (LiSat-11) is a meaningful outcome measure and may play an important role in setting rehabilitation goals in patients with chronic pain. Until now, there was no Japanese version of this questionnaire, and the psychometric properties of the original version of the LiSat-11 have only been investigated using classical test theory approaches. The objective of the present study was to evaluate the psychometric properties of the LiSat-11-J using Rasch analysis in a sample of Japanese people with low back pain (LBP).
One hundred and two participants with LBP completed the LiSat-11-J. Concurrent Validity of the LiSat-11-J was investigated by examining the relationships between the LiSat-11-J and clinical valuable. Rasch analysis was used to assess targeting, category ordering, unidimensionality, person fit, internal consistency, and differential item functioning.
The LiSat-11-J was significantly correlated with disability, pain-related catastrophizing, fear of movement, anxiety, and depression. The LiSat-11-J targeted the participants with low life satisfaction and had unidimensionality, good internal consistency, and good test-retest reliability. Responses to category 1 (“very dissatisfying”) and 2 (“dissatisfying”) were disordered. After collapsing these two categories into a single new category, targeting function and category order showed a little improvement.
The LiSat-11-J has unidimensionality, acceptable internal consistency, good test-retest reliability, and concurrent validity to the assessment of pain-related disability and psychological factors. However, targeting function and category order were not well. These two components showed improvement by rescoring category, but not sufficient. These results suggest that the LiSat-11-J is not well adequate to apply as an assessment tool of life satisfaction to the patients with LBP.
Rasch Analyses of the Wheelchair Use Confidence Scale for Power Wheelchair Users
2018, Archives of Physical Medicine and Rehabilitation
To examine the dimensionality of the Wheelchair Use Confidence Scale for power wheelchair users (WheelCon-P), to identify items that do not fit the Rasch rating scale model as well as redundant items for elimination, and to determine the SEMs and reliability estimates for the entire range of measurements.
Secondary analysis of cross-sectional data.
Community.
Volunteer participants (N=189) using wheelchairs (mean age of the sample, 56.7±13.0y; mean years of wheelchair use experience, 20.4±16.4).
Not applicable.
59-Item WheelCon-P.
Principal component analyses confirmed the presence of 2 self-efficacy dimensions: mobility and social situation. Eleven mobility items and 5 social situation items fit the Rasch rating scale model. Three items misfit the model using all 16 items (ie, WheelCon-P short form). In each of the mobility, social situation, and WheelCon-P short form range of measurements, the 2 lowest and 2 highest measures had internal consistency reliability estimates below .70; all other measures had reliability estimates above .70.
The WheelCon-P is composed of 2 self-efficacy dimensions related to mobility and social situations. The scores from the WheelCon-P short form and the 11-item mobility and 5-item social situation dimensions using a 0 to 10 response scale have good reliability.
Reducing Barriers to Patient-Reported Outcome Measures for People With Cognitive Impairments
2017, Archives of Physical Medicine and Rehabilitation
The field of rehabilitation has increasingly called for the use of patient-reported outcome measures (PROMs) in research and practice. Given that many rehabilitation patients present with conditions associated with cognitive impairments, it is imperative to reduce barriers to PROM use for this population. The purpose of this article is to develop a comprehensive understanding of cognitive accessibility that can prospectively inform the design of PROMs. We put forth the following definition of cognitive accessibility for PROMs: cognitive accessibility is present when assessment design anticipates respondent variability in cognitive abilities and, to the greatest extent possible, reduces cognitive demands and/or supports cognitive processes to enable respondents with a range of cognitive abilities to interpret and respond to assessment items as intended. Our operationalization of cognitive accessibility in measurement in the field of rehabilitation is informed by 2 assumptions: (1) cognitive accessibility results from an interaction between the individual's capacities and the demands of the assessment and assessment context, and (2) individuals with cognitive impairments have the right to be involved in decisions about their lives, including health care decisions. This article proposes 3 design features that can be optimized for cognitive accessibility: content, layout, and administration procedures. We end with a discussion of next steps that the field of rehabilitation measurement can undertake to advance our understanding of cognitive accessibility.
The use of concept mapping in measurement development and evaluation: Application and future directions
2017, Evaluation and Program Planning
Citation Excerpt :
Despite our observations, the direct effect of concept mapping upon the psychometric results found in these studies is unknown and we can only speculate on its impact. In general, the field has not yet systematically addressed the question of the degree to which qualitative methods improve the measurement qualities of a scale (Velozo, Seel, Magasi, Heinemann, & Romero, 2012). To do so would require a highly-controlled design that had a clearly established comparator method, a scenario that seems implausible and unlikely.
The past decade has seen an increase of measurement development research in social and health sciences that featured the use of concept mapping as a core technique. The purpose, application, and utility of concept mapping have varied across this emerging literature. Despite the variety of uses and range of outputs, little has been done to critically review how researchers have approached the application of concept mapping in the measurement development and evaluation process. This article focuses on a review of the current state of practice regarding the use of concept mapping as methodological tool in this process. We systematically reviewed 23 scale or measure development and evaluation studies, and detail the application of concept mapping in the context of traditional measurement development and psychometric testing processes. Although several limitations surfaced, we found several strengths in the contemporary application of the method. We determined concept mapping provides (a) a solid method for establishing content validity, (b) facilitates researcher decision-making, (c) insight into target population perspectives that are integrated a priori, and (d) a foundation for analytical and interpretative choices. Based on these results, we outline how concept mapping can be situated in the measurement development and evaluation processes for new instrumentation.
Systematic Reviews and Clinical Trials in Rehabilitation: Comprehensive Analyses of Publication Trends
2016, Archives of Physical Medicine and Rehabilitation
To analyze publication trends of clinical trials (CTs) and systematic reviews (SRs) in rehabilitation.
PubMed searches were performed with appropriate combinations of Medical Subject Headings. All entries until December 2013, and their yearly distributions since 1981 (when the first rehabilitation SR was identified), were retrieved. After the initial data visualization, data analyses were narrowed to specific periods. Linear regression techniques analyzed the growth of publications and their relative percentages over time.
Not applicable.
Not applicable.
Not applicable.
Not applicable.
Although not observed for SRs, CTs have grown at a much higher rate in rehabilitation than in the broader health/medical field—more than twice the difference for both periods analyzed (1989–2001, 2001–2013). Rehabilitation journals published about 20% or less of the rehabilitation SRs and CTs, and no significant increases were observed over time (P>.05; 2001–2013). Neurologic conditions, particularly cerebrovascular, were the most addressed by rehabilitation SRs and CTs, while differences between neurologic and other groups of conditions typically widened over time (eg, more than doubled between neurologic and musculoskeletal conditions in 15y).
While publications of CTs are increasing at a much higher rate within rehabilitation than within broader health care, further research is warranted to explain why this trend is not being followed by SRs, particularly those with meta-analysis. Similarly, research might determine whether the (growing) differences in the publications of rehabilitation SRs and CTs across groups of conditions are justified by clinical or population need.

View all citing articles on Scopus

: Supported in part by the National Institute on Disability and Rehabilitation Research, U.S. Department of Education (grant no. H133G080153).

: No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit on the authors or on any organization with which the authors are associated.

: Reprints are not available from the author.

View full text

Special communicationImproving Measurement Methods in Rehabilitation: Core Concepts and Recommendations for Scale Development

Abstract

Section snippets

Core IRT Concepts

Qualitative Methods

Qualitative Approaches

Summary and Conclusions

Arch Phys Med Rehabil

Phys Med Rehabil Clin N Am

Eval Program Plann

Value Health

Arch Phys Med Rehabil

Arch Phys Med Rehabil

J Clin Epidemiol

J Clin Epidemiol

Arch Phys Med Rehabil

Disability and rehabilitation research projects and centers program—Rehabilitation Research and Training Centers (RRTCs)—employment policy and measurement

Federal Register

Applying psychometric criteria to functional assessment in medical rehabilitation: IIDefining interval measures

Arch Phys Med Rehabil

The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years

Med Care

AHRQ Patient Centered Outcomes Research Institutional Award (K12) 2011

PCORI funding announcement pilot projects grants 2011

Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims

Qualitative research in occupational therapy: from the first to the second generation

OTJR (Thorofare N J)

Qualitative methodology for rehabilitation research

J Rehabil Med

PRO development: rigorous qualitative research as the crucial foundation

Qual Life Res

Mixed research as a tool for developing quantitative instruments

J Mix Methods Res

Evaluation of item candidates: the PROMIS qualitative item review

Med Care

Applying item response theory modeling for evaluating questionnaire item and scale propertiesAssessing quality of life in clinical trials: methods of practice

Observations are always ordinal; measurements, however, must be interval

Arch Phys Med Rehabil

Rasch and Wright: the early years

Rasch Meas Trans

Estimation of latent ability using a response pattern of graded scores

Psychometrika Monogr

Applying the Rasch model: fundamental measurement in the human sciences

Handbook of modern item response theory

Principles and selected applications of item response theory

Controversy and the Rasch model: a characteristic of incompatible paradigms?

Med Care

Computerized adaptive testing: a primer

Qual Life Res

Concept mapping for planning and evaluation

Special communication
Improving Measurement Methods in Rehabilitation: Core Concepts and Recommendations for Scale Development