The revised FLACC score: Reliability and validation for pain assessment in children with cerebral palsy

1 Introduction

Management of pain in children with cerebral palsy (CP) is problematic and due to the complexity of assessment often not adequately treated. Postoperative pain in children with CP is assessed more infrequently than in cognitively healthy children for which reason less analgesia is administered [1]. Management of postoperative pain, especially in the most affected children, is a concern, since a positive proportional relationship exists between the severity of CP, level of cognitive impairment and number of surgical procedures needed. Recent studies report, that a high proportion of children with CP have frequent episodes of pain resulting in decreased quality of life [2,3]. Until present study no validated pain assessment tools for children with CP have been available in Denmark.

The r-FLACC (revised Face, Legs, Activity, Cry and Consolability) pain score is an internationally acclaimed tool for assessing pain in children with CP unable to self-report because of its ease to use, its use of core pain behaviours and its clinical utility. In addition the r-FLACC pain score is superior to other pain assessment tools since the revision introduced an open-ended descriptor for incorporation of typical and atypical individual pain behaviours [4, 5, 6]. Children with cognitive impairment may have atypical pain behaviour due to idiosyncrasies masking the typical expression of pain among others laughing, singing, clapping of hands, anger, aggressiveness and self-injury [4,7,8]. The original FLACC score showed low subgroup reliability necessitating the revision. Subsequent reliability and validity testing of the revised score show excellent agreement in all of the 5 categories [4,9].

As with most of the validated pain assessment tools, the r- FLACC is developed in English and has not been validated following translation. Other pain assessment tools for children with CP validated for postoperative pain assessment include the Individualized Numeric Rating Scale (INRS) [10] with established construct validity and inter-rater reliability and the Noncommunicating Childrens’s Pain Checklist-Postoperative Version (NCCPC-PV) [11,12] with established construct validity, inter-rater reliability and internal consistency. Good psychometric properties were found for the Paediatric Pain Profile (PPP) [13] for assessment of everyday pain with established content and construct validity, inter-rater reliability and internal consistency and for the Echelle Douleur Enfant San Salvador (DESS) [14] for procedural pain assessment with established construct validity [15].

The quality of Health-Related Patient-Reported Outcomes (HR-PROs) is best described by three quality domains by the COS- MIN group [16]. Reliability (internal consistency, reliability and measurement error) estimates the extent to which scores for patients who have not changed are the same for repeated measurements under several conditions. Validity (content validity, construct validity and criterion validity) refers to the instrument actually measuring the construct it is designed to measure. Responsiveness refers to the probability of correctly identifying an adequate and valid reflection of changes between measurements [17, 18, 19, 20].

To the best of our knowledge the reliability and validity of the r-FLACC pain score have never been published by other than the developers. The aim of this study is to assess reliability and validity of the r-FLACC pain score for use in Danish children with CP.

2 Methods

The study was approved by the local ethics committee (M-20100189) and the Danish Data protection Agency (1-16-02-97-10). Oral and written informed consent was obtained from the parents of the children, and the study was carried out in accordance with the principles of the Helsinki Declarations. The clinical validation of the r-FLACC was undertaken during 2010–13. The translation of the r-FLACC complied with the international guideline set up by the Translation and Cultural Adaptation Group (TCA) – Principles of Good Practice (PGP) [21]. The COSMIN checklist was used as a guideline in the reliability and validity testing of the r-FLACC score. The checklist is designed for HR-PROs with high complexity and since the r-FLACC score is not patient-reported some discrepancies is present resulting in the COSMIN checklist being a guideline rather than an absolute method.

Twenty-seven children were included after informed consent was obtained. Inclusion criteria were children with CP who were not able to self-assess pain and planned orthopaedic surgery at the Department of Children’s Orthopedics, Aarhus University Hospital. The surgical procedure varied in severity from minor tendon surgery to major bony orthopaedic surgery.

Two methods for postoperative pain assessment were used at the same time in the postoperative hospital stay. The Observational Visual Analogue Score (VAS-OBS, range 0–10) was used at the bedside where the parents or primary caregivers estimated the pain intensity in their child during a 2-min period. A 2-min standardized video recording of the child was made in order to score the patient using the r-FLACC. In addition, in 10 of the included children, a preoperative baseline video-recording was made in order to assess construct validity of the r-FLACC score. In all the video recordings a close-up of the face, the legs and the total body of the child was performed. The parents or primary caregivers were asked to try to console the child if they found it necessary. Furthermore, the parents or primary caregivers preoperatively completed a questionnaire regarding the individual pain behaviours for their child related to each subgroup of the r-FLACC score.

Inter-rater reliability of the r-FLACC score was tested by review of the video recording by two registered nurses (RN) experienced in the care and pain management of children with CP. They independently assigned an r-FLACC score, using the translated version of the r-FLACC, after having familiarized themselves with the parent’s questionnaire of specific manifestations of pain for the child in question. Ten of the recordings were reviewed again 1 year later by one of the nurses in order to assess intra-rater reliability.

3 Results

The 27 children included were 3–15 years old, 11 girls and 16 boys with Gross Motor Function Classification Scores (GMFCS) scores ranging from II to IV. They all had tetraplegic CP and were not able to self-report pain. Twenty children underwent pelvic and femoral osteotomy, 4 children underwent tendon or soft tissue surgery, 1 child underwent calcaneal osteotomy, 1 child underwent epiphysiodesis of the distal femur and 1 child had femoral plates removed. The Danish version of the r-FLACC score was used for pain assessment. The original English version is presented in Table 1.

Reliability contains three measurement properties. Internal consistency was found to be excellent with a CA for the two raters of 0.9023 and 0.9758, respectively. The CA values for each subgroup are listed in Table 2. A factor analysis of the 5 subgroups showed low FAU indicating unidimensionality (FAU face: 0.55; FAU legs: 0.25; FAU activity: 0.31; FAU cry: 0.25; FAU consolability: 0.19).

Excellent intra-rater reliability was proved by a test-retest resulting in an ICC of 0.97530. Good inter-rater reliability between rater 1 and 2 was seen with an ICC of 0.74576. Both intra and interrater reliability are illustrated by Bland–Altman plots where the agreement is visualized (Figs. 1 and 2).

Fig. 1

Agreement (intra-rater reliability) illustrated by Bland–Altman plot (n =10) with comparison between the first r-FLACC scores from rater 1 and the re-test scores after 1 year. The differences between the pairs of measurements on the vertical axis are plotted against the average of each pair on the horizontal axis. The middle horizontal line reflects the mean difference (1.0 (95%CI:0.174; 1.826)) and the upper and lower line the limits of agreement (-1.309; 3.309).

Fig. 2

Agreement (inter-rater reliability) illustrated by Bland–Altman plot (n = 20) with comparison between rater 1 and rater 2. The differences between the pairs of measurements on the vertical axis are plotted against the average of each pair on the horizontal axis. The middle horizontal line reflects the mean difference (0.8 (95%CI: -0.119; 1.719) and the upper and lower line the limits of agreement (-3.126; 4.726).

No floor or ceiling effects were observed and the entire range of the r-FLACC score was used with a minimum r-FLACC score of zero in 30% of the children and a maximum score of ten in 7% of the children.

Validity contains three measurement properties. Content validity was tested by the originators of the r-FLACC. Construct validity was supported by a significant increase in r-FLACC scores following surgery compared to preoperative assessments (n = 17; difference 2.23; p = 0.0397). Criterion validity was established only in rater 1 by a Pearson’s correlation coefficient of 0.76 (n = 27) with rater 1 and 0.59 (n = 20) with rater 2 when comparing the r-FLACC scores of the two raters and the VAS-OBS scores.

4 Discussion

The current study establish excellent validity and reliability of the Danish version of the r-FLACC pain score for assessing postoperative pain in children with CP not able to self-report pain. In comparison to the original r-FLACC validation study, this study benefits from a homogenous study population only including cognitively impaired children, who were not able to self-report pain and by using the COSMIN checklist as a guideline. On the other hand, the study by Malviya et al. [4] benefits from a larger study population and includes both bedside and video assessments.

The internal consistency was found to be excellent for both raters but also for each item of the score assessed by both CA and a factor analysis confirming that all 5 items of the score are sufficiently correlated and measures the same construct, hence unidimensionality is established. However, Scholtes et al. [17] points out that internal consistency is established if CA values are between 0.70 and 0.95 and if above 0.95 the measurement instrument may contain too many items that are assessing the same underlying construct. This study finds a CA of 0.90 and 0.98 (two raters), which supports internal consistency. Since the r-FLACC is designed to measure pain as a very specific construct the high CA is expected and further establishes unidimensionality. This is opposite to other PROMs measuring more unspecific constructs, for example quality of life. The subgroup CA range from 0.86 to 0.98 with consolability having the lowest value for both rater 1 and 2, indicating that even though a high internal consistency, consolability is the item that least fit the score. The COSMIN checklist is developed for Health- Related Patient-Reported Outcomes (HR-PROs) often characterized by more items with higher complexity than in the r-FLACC pain score. Furthermore the r-FLACC is not a patient-reported outcome, but rather a tool for the clinician in measuring pain. Considering these issues a higher CA value than referenced for classical HR-PROs would be expected for the r-FLACC.

The intra-rater reliability in present study was excellent (ICC: 0.97530); but the inter-rater reliability only good (ICC: 0.74576). The Bland–Altman plots for both reliability measures illustrates that the differences of measurements are concentrated around the mean difference, though with a rather large interval of agreement for especially inter-rater reliability. This is in accordance to other studies testing the psychometric properties of pain assessment scales, where the range of ICCs for inter-rater reliability for the INRS was 0.65–0.87 [10], for the NCCPC-PV 0.78–0.82 [11] and for the PPP 0.74–0.89 [13]. This suggests that even though intra-rater reliability is established, the most exact pain assessment during hospitalization of a child with CP is performed using only one rater continuously for each child.

When developing an instrument as for example the r-FLACC all known relevant pain variables for this patient group have to be included in the instrument, hence content validity is a subjective assessment of the degree to which the r-FLACC adequately reflects the level of pain. The assessment of content validity of the r-FLACC score has been described by Malviya et al. [4] by a thorough review of pain behaviours common to individuals with cognitive impairment resulting in an expansion of descriptors in the least reliable categories of the original FLACC score (Table 1). This establishes content validity of the r-FLACC score and is transferable through translations.

In present study, construct validity was demonstrated by an increase in the level of pain after surgery. By definition construct validity estimates the degree to which the scores of the measurement instrument are consistent with predefined hypotheses on internal and external relationships [17]. In the case of pain assessment, it is a predefined hypothesis that the level of pain will increase following surgery showing an expected difference. Construct validity was also demonstrated by Malviya et al. [4] after revision of the original FLACC score by demonstration of a decrease in pain scores after administration of analgesics. The discussion between using agreement or reproducibility parameters is pointed out by de Vet et al. [24] and for a pain score like the r-FLACC agreement parameters could be more illustrative than reproducibility parameters. Nevertheless, in medical science, reproducibility parameters are more commonly used. Present study opted to calculate and report reproducibility by use of ICC similar to Solodiuk et al. [10] and in accordance to the COSMIN guideline. Opposite, Malviya et al. [4] calculates both kappa agreement and ICC reproducibility, making comparisons between the two studies difficult.

To assess criterion validity a comparison to another gold standard instrument with known good validity is a prerequisite. For the assessment of pain in children several instruments exists in the Scandinavian countries but none serves as a gold standard instrument. The Visual Analogue Score has been validated for use in several patient groups and is the mostly used pain score in Denmark and since no perfect gold standard for the evaluation of pain in children with CP exists, the VAS-OBS was chosen as an approximation for a gold standard instrument, knowing that it may be a limitation to the study. The r-FLACC did demonstrate acceptable criterion validity for rater 1 but not for rater 2, which may be caused by a number of factors. Firstly, Scholtes et al. [17] states that if the comparison instrument is not ‘really gold’, one would not know, which instrument is not valid if the comparison between the two instruments is low. Secondly, the VAS-OBS and r-FLACC scores have fundamental differences that might interfere with the comparison of the scores. One difference is that VAS-OBS involves assessment by parents and the r-FLACC involves assessment by a RN. Previous studies have found that parents tend to overestimate pain in comparison to other observers [4]. In addition, the r-FLACC score is assessed by use of a video-recording and the VAS-OBS score is assessed bedside. These factors might influence the comparison of the two instruments. In the study by Malviya et al. [4] criterion validity was determined by comparison to both the Nursing Assessment of Pain Intensity (NAPI), developed for healthy older infants and children [25] and for the children able to self-report by the Verbal 0–10 Numbers Scale or Simplified Faces Scale [4]. Some limitations are noted due to the NAPI not being a gold standard for pain assessment in children with CP and important demographic differences exists by comparing r-FLACC scores from children not able to self-report to a number or face scale from children self-reporting.

Assessment of r-FLACC scores via video-recordings is an advantage when estimating reliability, due to the fact that the level of pain of the child on the video-tape is unchanged and actual reliability estimates the extent to which scores for a patient who have not changed are the same for repeated measurements. The long interval is a benefit since it prevents recollection by the rater and this far outweighs the disadvantage that a possible change in the way the rater evaluates pain might occur.

The validation process was designed only to include children with postoperative pain and not procedural or chronic pain hence possibly limiting the generalizability of the r-FLACC pain score. However, by using a postoperative setting the risk of floor or ceiling effects was avoided since the entire range of the measurement instrument has been applied. In addition, no floor or ceiling effects were observed indicating that this measurement instrument is valid in measuring no, average and severe pain in children with CP.

5 Conclusions

This study benefits from a systematical approach to the validation and reliability parameters by using the COSMIN checklist as a guideline. Results showed excellent internal consistency, excellent intra-rater reliability and a good inter-rater reliability. In addition, excellent construct validity and criterion validity were established, though only for rater 1. Since a validation study of the original language revealed similar results it is evident that the r-FLACC pain score maintains its psychometric properties after translation. In conclusion, the r-FLACC pain score is both valid and reliable in assessing postoperative pain in children with CP not able to selfreport pain.

6 Implications

With the r-FLACC clinicians have a valid tool for assessing the level of postoperative pain, hence increasing the quality of pain management in children with CP. In addition the valid r-FLACC score has the potential for use in interventional research regarding pain management in this vulnerable group of patients. The r-FLACC is valid and reliable in measuring postoperative pain and future perspectives include validation of the r-FLACC score for procedural and chronic everyday pain and implementation into daily practice in the Scandinavian countries.

Highlights

• Management of pain in children with CP not able to self-report is difficult.

• Postoperative pain with atypical pain behaviours and idiosyncrasiesis common.

• We evaluated validity and reliability of the translated r-FLACC pain score.

• Excellent internal consistency and intra-rater reliability were found.

• Content, construct and criterion validity were established.