Development of a national audit tool for juvenile idiopathic arthritis: a BSPAR project funded by the Health Care Quality Improvement Partnership

Abstract

Objective

Timely access to holistic multidisciplinary care is the core principle underpinning management of juvenile idiopathic arthritis (JIA). Data collected in national clinical audit programmes fundamentally aim to improve health outcomes of disease, ensuring clinical care is equitable, safe and patient-centred. The aim of this study was to develop a tool for national audit of JIA in the UK.

Methods

A staged and consultative methodology was used across a broad group of relevant stakeholders to develop a national audit tool, with reference to pre-existing standards of care for JIA. The tool comprises key service delivery quality measures assessed against two aspects of impact, namely disease-related outcome measures and patient/carer reported outcome and experience measures.

Results

Eleven service-related quality measures were identified, including those that map to current standards for commissioning of JIA clinical services in the UK. The three-variable Juvenile Arthritis Disease Activity Score and presence/absence of sacro-iliitis in patients with enthesitis-related arthritis were identified as the primary disease-related outcome measures, with presence/absence of uveitis a secondary outcome. Novel patient/carer reported outcomes and patient/carer reported experience measures were developed and face validity confirmed by relevant patient/carer groups.

Conclusion

A tool for national audit of JIA has been developed with the aim of benchmarking current clinical practice and setting future standards and targets for improvement. Staged implementation of this national audit tool should facilitate investigation of variability in levels of care and drive quality improvement. This will require engagement from patients and carers, clinical teams and commissioners of JIA services.

Introduction

JIA is an umbrella term for a heterogeneous group of conditions characterized by chronic arthritis and categorized into sub-types according to clinical features at onset [1, 2]. JIA is one of the most common chronic inflammatory diseases of childhood with a UK incidence of 1:10 000 reported by Symmons et al. [3] in 1996. Although this equates to at least 1000 new cases per annum in the UK, the figure is almost certainly an underestimate; at that time, it was only possible to identify incident cases presenting to specialist paediatric centres. More recent incident data from other countries are higher [4, 5].

Appropriately tailored interventions are central to minimizing the adverse impact of JIA on physical, psychological and visual function, health-related quality of life and social/educational attainments [6]. However, there is wide variation in severity of presentation and disease course, and treatment-related morbidity can be significant. In part as a consequence of this heterogeneity, high quality evidence supporting best practice in JIA is limited, resulting in wide variations in service delivery.

The British Society for Paediatric and Adolescent Rheumatology (BSPAR)/Arthritis and Musculoskeletal Alliance Standards of Care (SOC) for Children and Young People (CYP) with JIA (2009) are consensus-derived minimum standards for clinical services delivering paediatric rheumatology (PRh) care [7]. The SOC were developed in accordance with national policy at the time [8, 9] and are endorsed by the Royal College of Paediatrics and Child Health. Key philosophies underpinning the standards included empowerment of patients and carers in treatment plans and a holistic approach to the provision of care. Despite the introduction of the SOC and an increasing awareness of the negative impact of delay in access to PRh care on disease outcomes in JIA [10], a 2013 multi-site UK audit against key SOC demonstrated considerable variation in service delivery and time to access specialist care [11]. In addition, the audit demonstrated a need for consensus agreed and measurable JIA-specific quality indicators, reflecting current clinical practice and including both clinician reported and patient/parent reported outcome measures, to enable standardization of clinical data collection. The 2013 audit formed the basis for a successful application to the Healthcare Quality Improvement Partnership (HQIP) to support development of the national audit tool for CYP with JIA. HQIP is an independent organization led by the Academy of Medical Royal Colleges, The Royal College of Nursing and National Voices and aims to promote quality in healthcare, in particular increasing the impact of clinical audit on healthcare quality improvement.

This paper describes the development of a nationally agreed tool and dataset for prospective national clinical audit of JIA. The audit tool will identify key service delivery measures necessary to benchmark current practice, and set future standards and targets for improvement; ultimately the aim is to reduce variations in delivery of care and improve health outcomes for CYP with JIA. To address the outcomes-based commissioning of National Health Service (NHS) services, a national clinical audit tool for JIA must include the UK NHS Specialist Commissioning for PRh specifications (NHS England Specialised Services Quality Dashboards) and also patient and carer reported outcome measures (PROMs) and patient and carer reported experience measures (PREMs).

In parallel to the development of this national audit tool for JIA, the UK PRh community has derived a standardized JIA-specific core dataset called Consensus derived, Accessible (information), Patient-focused, Team-focused, Universally collected (UK), Relevant to all and containing Essential data items (CAPTURE-JIA), to facilitate comparative clinical studies and research collaborations between providers of care [12]. The ultimate intention of the BSPAR community is to collect and embed the nationally agreed audit data items within CAPTURE-JIA, thereby incorporating indicators of service delivery, clinical outcomes, carer outcomes and experience into one dataset to be used in routine clinical practice to improve patient care.

The aim was to develop a national clinical audit tool for JIA that would enable evaluation of current clinical practice against both clinical outcomes and PROM/PREM. A secondary goal was to set future quality standards and to ensure these could be regularly assessed by standardized data collection with CAPTURE-JIA.

Methods

Development of governance structure

To ensure engagement with all relevant stakeholders (clinicians, specialist nurses, allied health professionals, academics and parents/patients), a robust governance structure comprising three expert groups was established as follows. A complex and consultative methodology underpins the development of this JIA audit tool (Fig. 1).

Multidisciplinary scientific steering committee (ScSC) (n = 14 including representation from all stakeholders)

The committee was responsible for overall operational management and final decision making. Membership was open to individuals engaged with development of BSPAR SOC, CAPTURE-JIA and/or HQIP application.

Multidisciplinary expert group (n = 21, including 11 members of the ScSC with representation of all stakeholders)

The group provided expert guidance with powers to recommend actions. The expert group invited expressions of interest from all NHS England PRh providers (identified through the NHS Providers List) and PRh representatives from Scotland and Wales. A purposeful sample of respondents was invited to participate in the study, with >80% of all UK sites represented. The sample included a range of service delivery models and geographic locations, a spectrum of track records of research from the National Institute for Health Research (and other research involvement), PRh national training centres (termed ‘grid centres’) and other paediatric rheumatology centres (termed ‘non-grid’).

PROM/PREM development group (n = 17)

This group was responsible for development of novel PROM and PREM questionnaires suitable for national audit. The BSPAR National Parents Group was established in 2013 to identify and represent the experiences of families affected by musculoskeletal diagnoses, most commonly JIA. The PROM/PREM development group included the BSPAR Parents Group (with representatives from across England and Scotland), invited consumer representatives from the National Institute for Health Research, Clinical Research Network Children and Arthritis Research UK Clinical Studies Group and young people with JIA.

Domains of audit

The ScSC established four domains for the audit, namely inclusion criteria and patient eligibility, service delivery quality measures and two aspects of impact, physician reported outcome measures and PROM/PREMs.

Inclusion criteria and patient eligibility

The ScSC agreed inclusion criteria and eligibility. Discussion focused on the time window for collection of audit data, follow-up time per patient and decision to include only new patients or all patients attending rheumatology outpatients within a given audit time frame. Advice was taken from the Chair of the British Society for Rheumatology (BSR) Clinical Affairs Committee, which is responsible for the National Clinical Audit for Rheumatoid and Early Inflammatory Arthritis (BSR EIA) [13].

Quality measures

An iterative approach between the ScSC and expert group was taken. The ScSC derived a preliminary long list of possible questions addressing key quality measures based on the BSPAR Standards of Care, the Scottish Paediatric and Adolescent Rheumatology Network Quality Standards, clinical commissioning requirements in England (NHS England dashboard for PRh) and additional suggestions from the wider ScSC. The specific data items required for each quality measure were listed and duplicate data items were removed to form a single list.

The feasibility of collecting the proposed list of data items was discussed with the expert group, highlighting the need for further prioritization. The ScSC prioritized the quality measures on the basis of perceived importance, inclusion in the commissioning dashboard and audibility (i.e. ease of collection of the necessary data items or overlap with CAPTURE-JIA). The expert group were asked to approve the final list.

To understand the impact of variation in service delivery, the quality measures will need to be assessed against the most important clinician reported outcomes and PROMs/PREMs.

Clinical outcome measures

The ScSC agreed that the primary outcome measure would be a measure of disease activity and worked in consultation with the Clinical Studies Group JIA Topic Specific Group to develop a list of candidate disease activity measures informed by a literature review of disease activity assessment tools [14] and an audit of disease activity assessment in clinical practice [12]. All recognized PRh NHS England providers and representatives from Scotland and Wales were invited to participate in the audit.

At a facilitated meeting of the expert group, a three-round nominal group voting exercise was used to achieve consensus (majority agreement of ⩾70%) on the primary disease activity measure for inclusion. A consensus definition of 70% was selected to align our work with consensus definitions favoured by the OMERACT collaboration [15] and the Core Outcome Measures in Effectiveness trials initiative [16]; the consensus cut-off of 70% has been used in a number of previous paediatric rheumatology studies [17–19]. Following approval by the ScSC, secondary measures were agreed via an online consultation process with the expert group.

PROMs/PREMs

Inclusion of PROMs and PREMs was a core requirement of the project. The PROM/PREM development group led the development and selection process, using a staged approach.

Patients and parents attended a facilitated workshop in which key outcomes and experiences for CYP with JIA were identified. The open exchange workshop was moderated by a PRh Consultant (N.W.). Key messages were brought together during and after the workshop by one of the consumer members of the BSPAR Parents Group (S.D.).

Identification of broad themes and sub-themes

Similar items were collapsed into a single item and grouped into broad themes and sub-themes by a sub-group of the ScSC. These were approved by the BSPAR Parents Group.

Comparison with existing validated questionnaires

Any pre-existing validated questionnaires capturing any of the broad themes or sub-themes prioritized by the patient/parent group were considered for the audit. Factors considered included length of questionnaire, ease of scoring and the time referent period used.

Online prioritization of broad themes and sub-themes

A simple Yes/No online consultation exercise was undertaken with all members of the PROM/PREM development group to determine whether they would prefer one broad question capturing several aspects of a theme, or one specific question addressing just one sub-theme.

Questionnaire development workshop

A second facilitated workshop was held to finalize the development of the PROMs and PREMs. The goals for this final workshop were to agree the number of patient outcome and patient experience questions for inclusion, agree the time reference period for questions (e.g. today, in the last week, in the last month) and develop wording for the questions and the format for the answers, ensuring that the questions were (i) generic (i.e. transferrable to other paediatric musculoskeletal conditions), (ii) relevant and valuable to all patients/parents and (iii) suitable for inclusion in CAPTURE-JIA to facilitate clinical adoption.

Following the meeting, a further online exercise with all members of the patient/parent group was undertaken for refinement and final approval of the HQIP audit PROM/PREM questionnaires.

Testing face validity of the questionnaire

Two additional groups of parents attending one of two JIA family days in April 2016 (group 1: 20 parents attending a Scottish Network for Arthritis for Children weekend in Scotland; group 2: 11 parents attending a JIA family day in Oxford) were asked to complete the PROM/PREM questionnaires and an additional brief questionnaire including questions on completion time, ease of completion and ease of understanding. A group discussion to highlight any key issues was facilitated by one of the research team.

A full list of data items and an associated data dictionary were prepared.

Results

Membership of each of the three expert groups is provided online (supplementary Table S1, available at Rheumatology Online).

Inclusion criteria and eligibility

JIA is an uncommon disease with relatively low numbers of new patients presenting year-on-year. Therefore, to ensure capture of sufficient data for meaningful analysis within a reasonable audit time frame, the ScSC agreed that all JIA patients (new and follow-up) be included.

Quality measures

The ScSC identified 14 potential quality measures requiring the collection of 40 data items. Overall the expert group felt that the data required for this would be too extensive in the absence of an electronic data capture system. The ScSC therefore agreed to prioritize 11 quality measures requiring 32 data items, including all those that map to the commissioning questions (Table 1). The expert group approved the final list.

Clinical outcome measures

A literature review informed the identification of 19 candidate disease activity measures (12 single and 7 composite measures) by the ScSC in collaboration with the JIA Topic Specific Group. A subsequent audit of current practice was completed by 10/14 invited PRh providers (71%), with data available for 153 children (median age 12 years, range 1–21 years). Although there was widespread agreement around important single measures of disease activity (100% centres intended to collect the core outcome variables, pain and uveitis), collection and documentation of these measures was inconsistent and unreliable, ranging from 87% (active joint count) to 43% (presence/absence of active uveitis) [12].

This information informed the expert group workshop. Although some members of the expert group had completed the audit, the expert group included a wider spectrum of centres and non-clinician stakeholders. The expert group achieved consensus (>70% agreement) on the three- or four-variable Juvenile Arthritis DAS as the primary outcome measure [20, 21] and presence/absence of uveitis (Y or N) as a secondary outcome.

The ScSC approved the choice of outcome measures but following some discussion, a simple yes/no consultation process with the expert group led to two modifications; documentation of presence/absence of sacroiliitis in enthesitis related arthritis and inclusion of the systemic Juvenile Arthritis DAS composite measure when published. The final list of clinical outcome measures is given in Table 2.

PROMs and PREMs

The initial PROM/PREM development group workshop initially identified 40 outcomes and 63 experiences. The outcomes and experiences were subsequently collapsed into three broad themes with 13 sub-themes for PROMs and six broad themes with 17 sub-themes for PREMs (Table 3).

For PROMs, some existing questionnaires were identified that captured many of the same themes, for example, Child Health Utility 9D Index (CHU-9D) [22]. However, these were all considered too long or the scoring system too complicated for use in an audit. In addition, the referent time point form most questionnaires was today only; the group considered this too short a time frame to provide useful information.

For PREMs, just two pre-existing tools were identified: The Friends and Family questionnaire [23] and the PREM developed as part of the BSR EIA [24]. The Friends and Family questionnaire was considered to be too generic. The adult inflammatory arthritis PREM has many overlapping domains and could potentially have been adapted; however, once again the group considered this to be too long.

A prioritization exercise was sent to the PROM/PREM development group to identify broad themes and sub-themes for inclusion and there was an 88% (15 people) response rate. At the start of the second workshop, those attending (n = 8) agreed a cut-off of 11/15 (73%) to be the minimum requirement for inclusion of each theme in the audit. On initial review, no single broad theme or sub-theme met these criteria. Discussion continued and the group achieved consensus (>73%) within the meeting, enabling development of appropriate question and response options for each theme. Once finalized, the questions were sent to the wider parent/young person group (initial 15 respondents), and consensus was once again achieved.

Given that JIA is a relapsing and remitting condition that can vary widely over relatively short time periods, a referent period of 1 month was agreed as sufficient to capture variability but short enough to minimize recall bias.

Following the workshop, the final set of questions was sent to all members of the PROM/PREM development group and minor modifications made based on feedback. The final PROM and PREM against which the quality measures will be assessed are shown in Table 2. The PROM/PREM questionnaires are shown in Table 4.

Feedback from the two additional family groups (Scotland and Oxford, April 2016) was very similar. Average time to complete the questionnaire was 3.44 (s.d. 1.49) min for group 1 and 3.45 (s.d. 1.51) min for group 2; 75% and 100% of groups 1 and 2, respectively, reported the questionnaire to be ‘easy’ or ‘very easy’ to complete and 85% and 91%, respectively, were ‘happy’ with the wording used.

Final list of audit data items and associated data dictionary

The full list of data items (n = 32) plus the PROM/PREM questionnaire (n = 7) is shown in Table 5. The associated data dictionary that would need to be collected to quantify the measures and the outcomes is provided online (supplementary Table S2, available at Rheumatology Online).

Discussion

This report describes the highly consultative consensus-based methodology underpinning development of a JIA-specific national audit tool including identification of key quality measures for inclusion, and selection of optimal clinician and patient/parent reported outcome tools to understand the impact of variations in service delivery.

National audit programmes aim to improve disease outcome for patients. In the UK, the Royal College of Paediatrics and Child Health supports national audit programmes in paediatric diabetes, neonatal care and epilepsy [25]. Such programmes provide an overview of disease related outcomes and provide potentially powerful benchmarking tools. In diabetes care there is some evidence that investment in regional networks and the introduction of a Best Practice Tariff mandating participation in audit has resulted in some improvements in outcome, but this is still relatively limited and also variable geographically [26]. There are synergies between all chronic diseases of childhood—fundamentally improving the quality of care for children is not only important to control symptoms and alleviate distress, but will also reduce lifetime risk from complications of the disease. A further synergy is with inflammatory arthritis in adulthood. The first national clinical audit for rheumatoid and early inflammatory arthritis highlighted geographically variable gaps between National Institute for Health and Care Excellence standards and existing care [13]. It is reasonable to hypothesize that the same gaps are likely to exist in the provision of paediatric care.

Implementation of a national audit programme for JIA will require a staged approach. The first step will be an initial data collection exercise generating benchmarking data against which robust and validated quality standards can be set. This will facilitate effective quality assessment for NHS England commissioning of specialized services. The second data collection phase will assess how well the standards are being met in terms of the quality, safety, outcome and experience of care for CYP with JIA.

Future participation of all PRh NHS Providers in a national clinical audit process evaluating the care of CYP with JIA will capture a more complete impression of what is happening at local (Trust), regional and national levels. The completion of the audit tool could in itself be considered a quality measure for the delivery of care in JIA. Collecting audit data in itself (challenging though this is) is simply not enough—the data must be utilized to actually deliver the change required to drive quality improvements and this requires concerted effort and commitment from all stakeholders’ services and access to fit-for-purpose IT systems.

Describing and understanding current clinical practice through national clinical audit will enable the development of robust and meaningful quality standards that will be used to drive improvements in clinical care and outcomes for JIA. Important national audit questions and quality measures may change over time. In particular, the current audit tool does not include an assessment of damage. Clinical and radiological evidence of damage occurs relatively late in JIA and damage assessment tools are not currently used routinely in clinical practice. Although a valid and meaningful assessment of damage could not be included in the audit tool at this time, damage may merit inclusion in future iterations, as damage assessment tools evolve further.

The active participation of the wider PRh clinical and research communities across the UK is an important strength of this study. It demonstrates the widespread support for projects informing best practice, the need for greater collaboration between centres, and support for a consensus process to agree and define a feasible dataset (i.e. set of data items to be collected) to encompass the needs of patients and families, clinical teams, research teams, service providers and commissioners. Although we acknowledge that our primary aim of assessing all patients with new onset JIA may be aspirational, the active participation of the wider PRh community in this and other national projects suggests that identification of all new patients could be a realistic expectation.

Patient experience and outcomes are key quality indicators but few condition-specific paediatric PREMs are available. A further strength of the study is the methodologically sound development of novel PROM/PREM tools specific to JIA and developed in consultation with the BSPAR Parents Group.

A major challenge for the PROM/PREM development group was achieving agreement on which broad themes and sub-themes to include. While agreement was achieved, a number of important sub-themes have been excluded. Future development of an electronic data collection tool may enable inclusion of further data items. The PROM/PREM tools require further validation and piloting prior to incorporation into standard clinical practice.

A consensus agreed core dataset for JIA (CAPTURE-JIA), which incorporates the audit data items, has been developed and is currently being piloted. Once finalized, embedding the collection of CAPTURE-JIA data items into routine clinical care would facilitate a national audit programme.

The longer-term goal is development of an electronic data collection tool with routine integration of data collection into electronic patient records. Improved documentation (and auditing) of service delivery at local (trust), regional (clinical networks) and national level against these important clinical and consumer reported outcomes will likely identify potential areas for healthcare quality improvement, key to improving clinical outcomes for this vulnerable group of patients.

Implementation of the proposed national audit tool should facilitate investigation of variability between services and across networks, identify current levels of care and drive a long-term quality improvement programme for children and young people with JIA. Healthcare providers designated as providing paediatric rheumatology services should be held accountable to address inadequacies in care and recommendations when demonstrated using the validated audit tool.

Acknowledgements

The authors thank the British Society for Paediatric and Adolescent Rheumatology (BSPAR) membership for their active support and contributions throughout this process. In particular, the authors thank the BSPAR Parents Group led by Mrs Sharon Douglas with the support of Dr Nick Wilkinson, Dr Nicola Smith for her support with the PROM/PREM validation work, Dr Allison Worth, Patient and Public Involvement Advisor Edinburgh Clinical Research Facility, for her support with the initial facilitated workshop and the JIA Topic Specific Group (affiliated to the National Institute for Health Research Clinical Research Network (NIHR CRN) Children and Arthritis Research UK Clinical Studies Group) for their support identifying and selecting suitable disease-related outcome measures. W.T. is supported by Arthritis Research UK grant number 20380 and the National Institute for Health Research Manchester Musculoskeletal Biomedical Research Unit. The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, The National Institute for Health Research or the Department of Health.

Funding: This work was supported by a grant from the Healthcare Quality Improvement Partnership (HQIP NCA 119) and a grant from Arthritis Research UK (20877).

Disclosure statement: K.B. received support from Abbvie and Pfizer for educational events and team away day. All other authors have declared no conflicts of interest.

Supplementary data

Supplementary data are available at Rheumatology Online.

References

Author notes