Article Text
PDF
Abstract
Objectives To develop and validate a new allergy-specific patient-reported experience measure (PREM) for children and their parents, and to collect feedback in an integrated care setting.
Design Two allergy-specific PREMs were produced using focus groups, cognitive testing, two prospective validation studies (collaboration: Royal College of Paediatrics and Child Health, Picker Institute Europe, Imperial College/London): ‘Your Allergy Care’, for children aged 8–16 years; ‘Your Child's Allergy Care’, for parents of children aged 0–7 years.
Setting Community event, primary/secondary/tertiary allergy care settings.
Main outcome measures Performance of PREMs in validation study; reported experience of allergy care.
Participants 687 children with allergic conditions and their parents/carers.
Results In total, 687 questionnaires were completed; 503/687 (253 child; 250 parent) for the final survey. In both surveys, demographic variations were not associated with differences in results. Although 71% of patients reported one or more allergic conditions (food allergy/eczema/hay fever/asthma), 62% required multiple visits before receiving final diagnosis. Overall, patient experience was good for communication with patient/parent, competence and confidence in ability, and 73% felt looked after ‘very well’ and 23% ‘quite well’. Areas for improvement included communication with nurseries/schools, more information on side effects, allergic conditions and allergen/irritant avoidance. Allergy care in primary/emergency care settings was associated with higher problem-scores (worse experience) than in specialist clinics.
Conclusions These new PREMs will allow allergy-specific patient experience reporting for children and parents and help identification of priority areas for improvement and commissioning of care. Efforts towards better allergy care provision must be targeted at primary and emergency care settings and underpinned by improving communication between healthcare providers and the community.
- Allergy
- Children's Rights
- Health Service
- Patient perspective
- Qualitative research
This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/
Statistics from Altmetric.com
What is already known on this topic
Patient-reported experience measures (PREMs) are important for evaluating care but very few condition-specific PREMs are available and even fewer for children.
Children are able and willing to report their own experience of care, and it often differs from that of their parents.
Allergy care for children is difficult to access and provision is patchy despite it being the most common chronic childhood condition in the UK.
What this study adds
This study presents the first disease-specific PREM for children with allergic conditions, an essential tool and key quality indicator for future planning and commissioning of services.
One-third of children with allergic disease have to visit four to eight healthcare professionals before receiving a diagnosis and complete treatment, suggesting significant training needs.
Information needs are well described but frequently not met, offering an immediate area for improvement of children's allergy care.
Introduction
Allergic conditions affect up to a third of children in the UK at some point during childhood making them the most common chronic childhood ailment.1 Many children suffer from multiple allergic comorbidities, such as eczema, asthma, food allergies and hay fever. Access to specialist allergy care is difficult due to underprovision of services, as outlined in successive national reports.2–5 Once accessed, there is evidence of considerable variability in the quality of allergy care, which is likely to negatively impact on clinical outcomes, patients' experiences and safety.6 ,7
In response, the Department of Health commissioned the Royal College of Paediatrics and Child Health (RCPCH) to define care pathways for children with allergic conditions, including the development of allergy-specific patient-reported experience measures (PREMs).8
PREMs allow patients to report on their experience of care rather than health outcomes (patient-reported outcome measures (PROMs), eg, quality of life tools).9 Clinicians, service providers and commissioners can and should use PREMs to collect feedback as a key tool for evaluating current care and for shaping future service provision.10 ,11 Although there are young-patient feedback tools that are generic to services,12–15 few are condition-specific, and the aim was to create a tool developed by children for children and by families for families to measure allergy-related patient and parent experiences of care.
Methods
The PREMs were developed collaboratively by the RCPCH Allergy Care Pathway Project Team (for the stakeholders’ list, see ref. 8) and the Picker Institute Europe using established survey-development methodology (figure 1).16 ,17 In this report, the term ‘allergy care’ encompasses any care for allergic conditions, delivered in primary, secondary and/or tertiary care, as well as emergency care settings, and the term ‘parent’ is used for parent/carer.
Flow chart of patient-reported experience measure (PREM) development.16 ,17 ,20 Patient-reported outcome measures (PROMs, eg, quality of life tools) have set/defined ways of validating instruments. PREMs need to be distinguished from PROMs where there are no pre-existing ‘criterion’ measures against which a new instrument can be correlated. Validity in this context is seen as fitness for purpose. A large part of the validation in PREMs is in cognitive testing, which was done repeatedly for the PREM in this study. Survey validation is an ongoing process of weighing the evidence. The question interpretation and response processes were assessed using cognitive testing to ensure that people understand the questions as intended and can respond accurately. Once data had been collected, further validation statistics were explored to evaluate whether the data were of adequate quality and delivered useful information. These analyses can be used to determine how well a new survey functions and to advise whether any questions should be considered for removal. The validation process explored whether completion rates were adequate for all questions and if there was any evidence of drop-out as respondents progressed through the questionnaire. Item completion rates were good for both questionnaires, and there was no evidence of fatigue. The scoring system (problem-scores) demonstrated good capability of showing discrimination. Therefore the overall tools functioned well and no amendments were required.
Sequential steps of PREM development, validation and testing
Focus groups
Allergy care service users (children (8–11 years), young adults (15–21 years), parents of allergic children (0–11 years); see online supplement 1) were invited to give their views on their allergy care; this was aided by an experienced facilitator and interview topic guide. Main themes were identified and formed the basis of question development. Two PREM surveys were developed: one for completion by parents of children aged 0–7 years and the second for completion by children aged ≥8 years, the latter with an additional shorter parents’ section.
Supplemental material
Cognitive testing
This iterative process tests understanding of each version of the survey among the target population (ie, young allergy patients/their parents) and ensures they can respond accurately. Cognitive interviewing refines a survey tool and is repeated until the wording, comprehension and volume of questions are fit for purpose.
Pilot study (version 1)
The initial pilot survey was in-field for 8 weeks in a tertiary allergy centre. Trained staff handed out the surveys (hand-out methodology18 ,19) and were available to support participants during completion.
Validation study (final version)
The refined PREMs, ‘Your Allergy Care’ (child survey: 22 child questions; 13 parent questions; free text comment box) and ‘Your Child's Allergy Care’ (parent survey: 34 questions; free text comment box), were handed out at a community event in primary/secondary/tertiary care clinics.
Analysis (each version)
The survey data were analysed using Picker ‘problem-scores’. These show the percentage of patients for each question whose response indicates that a particular aspect of their care could have been improved. This summary measure can assist with prioritising areas for improvement. Problem-scores are calculated by combining the non-optimum response categories, for example, if a question is answered with ‘Yes’, 60%; ‘Yes, sometimes’, 25%; and ‘No’, 15% the problem-score will be 25%+15%=40%. Large scores highlight potential problems, and low scores reflect better performance.
Statistical analysis
The sample size (250 completed of each survey) is based on evaluation of tetrachoric correlations, which are appropriate for problem-scores.20 The required sample size to detect a correlation of at least 0.3 is 250 (α=0.05, 1-β=0.8). Due to the self-completion method, missing responses are inevitable and percentage calculations exclude respondents that did not answer. Item frequencies were examined for the proportion of missing or non-evaluative responses, to detect early drop-out from the questionnaire and to identify potential ceiling effects. Item variances were examined to ensure questions provided sufficient score differentiation.
Interitem tetrachoric correlations were computed to examine the relationship between questions and the overall structure of the questionnaire. Correlations were computed between each evaluative question and the ‘overall rating’ question. Score variation was investigated using general linear models with gender, ethnicity and who completed the questionnaire (child or parent) as main effects. Wherever appropriate, level of association (Kendall's τ-b) was calculated between corresponding parent and child responses.
The project falls under service evaluation, approved by the relevant institutional review board (Imperial College Healthcare NHS Trust)—it was not ‘research’, that is, not exploring specific hypotheses but instead asking people to provide feedback on their care. Therefore, ethical approval was not sought. The project work conformed to the Market Research Society's Code of Practice, for example, informed consent was obtained from all cognitive interview and focus group participants prior to their involvement.
Results
PREM development, testing and refinement
Focus groups: qualitative interviews
Three focus groups were held involving children and young adults with a range of allergic conditions (asthma/eczema/rhinitis/food allergies/anaphylaxis) and their parents (figure 1). The majority of children had one or more allergic condition. Four main themes emerged (table 1; see online supplement 1).
Main themes emerging from focus group interviews
Question development, refinement and pilot study of survey V.1 (V1)
Two surveys were developed:
V1 Child survey ‘Your Allergy Care’ with short parent section (89 questions)
V1 Parent survey ‘Your Child's Allergy Care’ (89 questions)
These were cognitively tested with five parents and five children (aged 2–14 years), amended accordingly and then completed by 74 children (and parents) and 110 parents attending a tertiary allergy clinic. Interim validity-testing performed on this sample, despite small numbers, revealed the following for the child (n=74) and parent (n=110) V1 surveys:
a large number of questions had been missed out and
correlations showed a wide range of relationships between items: some highly and some negatively correlated.
Key findings from this first pilot V1 survey include: Children mostly approached their general practitioners (GP, 51.5%) or emergency departments (24.3%) for their first allergy care. While 38.5% of parents felt that their GP did not know enough about their child's allergic condition, 71.3% of parents and 85.7% of children rated their/their child's overall allergy care (any care setting) as good/excellent, and 6% of the parents and 1.4% of children as poor/fair. Communication between healthcare professionals (HCP) was rated not very good/not at all good by 20.2%; this increased to 28.4% for communication between HCP and nursery/school. Moreover, 61.5% (64/110) of parents reported that their child had an action plan but only 3 had it drawn up in primary care.
The surveys were refined based on interim analysis results and with input from stakeholders (questions trimmed, clarified, duplicates removed; number of questions brought down from 89 to 55, and finally 35). Further cognitive testing (seven parents; five children) was performed on the final version, resulting in:
Child survey: 22 child questions, 13 parent questions; free text comment box
Parent survey: 34 questions, free text comment box.
The final surveys are included as online supplement 3.
Supplemental material
Final PREM validation and testing in various community and healthcare settings
Final survey: demographics, allergic conditions and validation results
The final surveys include four domains mapping to focus-group themes (table 1) with the headings: ‘Diagnosis’, ‘Seeing HCPs’, ‘Managing your/your child's condition’ and ‘Lifestyle’. Furthermore, they cover information about ‘Your/your child's allergy’ and ‘Just a bit about you’ (demographic information). A total of 503 surveys were completed (parent survey, n=250; child survey, n=253; parent section incomplete for 13/253 (5%)). Table 2 shows demographic and clinical background information of the survey respondents.
Demographic information, allergic conditions
The majority of patients had long-standing chronic disease and a third of respondents had to visit HCPs four or more times to obtain a diagnosis.
Demographic variations were not associated with differences in results in either survey, non-response rate was low (<5%), there was no sign of early drop-out and questions showed good discrimination overall. Questions relating to ‘how well looked after by most recent HCP’ and ratings of overall care had low problem-scores, indicating a ceiling effect in this format, that is, they were not good at discriminating ‘good’ and ‘poor’ experiences as they were answered mostly positively. There was moderate agreement (Kendall's τ=0.39–0.43) between specific items in the child survey and child survey parents’ section (see online supplement 2).
Supplemental material
What do our patients and their parents/carers tell us?
The results from parent and child surveys were moderately similar for most items. Figure 2A, B summarises results and problems-scores. Items where parent and child responses varied notably are presented separately within figure 2A, B.
(A) Patient experience reported for all settings (Domains: seeing HCP, most recent; emergency plan & treatment; information). Survey questions/answers presented for all respondents, where indicated results were separated into parent and child responses. Problem-scores are a summary measure to represent the proportion of respondents to a particular question who identified that there was room for improvement. Lower scores reflect smaller problems, whereas higher scores are indicative of poor performance. (B) Patient experience reported for all settings (Domains: communication; support; decisions; overall care). Survey questions/answers presented for all respondents, where indicated results were separated into parent and\r\nchild responses. Problem-scores are a summary measure to represent the proportion of respondents to a particular question that identified that there was room for improvement. Lower scores reflect smaller problems, whereas higher scores are indicative of poor performance. (C) Patient experience reported in tertiary versus other care settings. Survey questions/answers presented for all respondents in each setting, but only items with ≥20 responses analysed. A&E, Accident & Emergency; HCP, healthcare professional.
Overall, patients/parents felt that they and their conditions were well looked after by HCPs. Areas for improvement (high problem-scores) were:
communication (i) among HCPs and (ii) between HCPs and nursery/school;
information giving, in particular on conditions themselves, allergen/irritant avoidance and medication side effects; and
access to support, such as a phone number to contact HCPs and/or information on support groups.
Parents expressed more interest in support groups and information on emergency treatments in the acute care setting than children (figure 2A, B).
Participants who attended tertiary allergy clinics reported better experience than those who attended other healthcare settings for their allergy care (figure 2C). In comparison, problem-scores were much higher in primary and emergency care settings.
Participants' free text comments from open-ended questions were converted into word clouds providing visualisation of the words used and give further insight into current experiences of allergy care (figure 3A, B).
(A) Free text comments on allergy care—examples. (B) Word clouds* using free text responses from both surveys. *Word clouds perform a count on the frequency of words to produce an image. The more mentions a word gets, the bigger the size of the word in the image. All words are adjusted proportionately and placed randomly in the cloud.
Discussion
Patients' experiences, along with outcomes, are at the heart of healthcare provision and there is increasing national emphasis on PREMs being used by service users, service providers and commissioners as key quality indicators.11 ,21 However, while there are many PROM tools for children,22 ,23 there are few validated PREMs, and these tend to assess children's experiences of a specific hospital visit (eg, the Picker Institute's acute paediatric surveys).
We present the first disease-specific, PREM for children with allergic conditions and report on experiences that children and their parents/carers have had when seeking and receiving allergy care in a variety of community and NHS healthcare settings.
Limitations
The validation study was carried out in multiple settings where children receive allergy care. Due to staffing constraints, the majority of PREMs were completed in a tertiary care centre (74%). It is likely that high satisfaction with care and specific HCPs is related to this, as families benefitted from an expert multidisciplinary team with generous time allocation. Comparison between tertiary and primary/community care confirms this, showing high problem-scores across domains for non-tertiary healthcare provision.
The number of surveys handed out was not recorded (multiple staff; multiple settings), which is a recognised drawback of hand-out methodology. Surveys were handed out and participants were encouraged to complete them ‘there and then’. The estimated return rate was >70%, based on copies produced and wastage. Hand-out methodology or point-of-care administration is associated with higher satisfaction scores compared with postal questionnaires,18 ,19 ,24 making positive response bias possible. The free text responses, collated in word clouds, reveal more clearly where patients experienced difficulties.
London has an ethnically diverse population, and surveys were available only in English. Demographic information on >90% of PREM respondents reflects the diversity of the local patient population (2011 Census; London-Westminster resident ethnicity profile: white British/other, 60%; black, 5.5%; Asian, 11.9%; mixed, 5.2%; other, 13.7%25). Although more than two-thirds of children in North West London have English as their second language, they attend English-language schools and questionnaire completion did not significantly differ between ethnic groups. However, it is likely that families with poor English-language skills will have worse experiences in accessing allergy care and therefore will be under-represented in this study.26 It is therefore important that PREMs (ideally in relevant languages) are used in community settings, especially in inner city areas with high deprivation indices, to elucidate children's experiences there.
Rationale for the PREM: does it achieve its objective?
This PREM was developed by children for children and by families for families. It performs well in a variety of community and healthcare settings. A total of 162 children completed the questionnaires by themselves with low drop-out or missing data, suggesting that the surveys are appropriately constructed and children can and will use them. While parent and child responses appeared mostly congruent, some differences were apparent and need to be taken into account.27
What are the messages from the first children's allergy PREM study?
The overall messages are the same for children and parents/carers. Taking the highest 25% of problem-scores (reflecting worst experience), key areas for improvement include:
Access to care, as many children have to visit numerous HCPs before getting a diagnosis
Communication involving HCPs: HCP to HCP across healthcare settings; HCP to nursery/school
Information about side effects, triggers/allergens and their condition overall
Support via patient support groups and directly from HCPs via a contact phone number
Problems with accessing allergy care are well recognised2 ,3 emphasising a lack of allergy services in the face of huge demand.1 Although 90% of children with asthma and allergies should receive treatment in primary care, this is not happening owing partly to lack of education in allergy during undergraduate and postgraduate training.28 A survey of GP trainers identified that few felt competent in allergy diagnosis and treatment,6 explaining the lengthy journey families have until their child is diagnosed. The situation is compounded by the pressure on primary care providers to be gate-keepers to specialist (secondary or tertiary) referrals29 and by the lack of specialist services, leaving few referral options. Our data confirm the knowledge gap, with higher problem-scores in primary/emergency care-settings compared with specialist clinics. Initiatives such as the RCPCH care pathways8 ,30–34 and ‘Itchy-Sneezy-Wheezy’ integrated care project (http://www.itchysneezywheezy.co.uk) with community-based allergy clinics and educational initiatives to improve competencies in all care settings are a step in the right direction as a recent report on general paediatric community clinics confirms.35
Our study confirms problems around communication and information giving/sharing.
However, some of the apparent unmet information needs may relate to poor patient recall after healthcare consultations.36 Information provided on causes, treatments and avoidance strategies in relation to different allergic conditions is frequently complex and HCPs must ensure that they offer opportunities to increase uptake of information via web-based resources, phone apps and so on as well as clinic letters, including treatment plans, written in appropriately adjusted language.
Communication with nurseries/schools requires permission from patients/parents.37 Including standardised personalised management plans (eg, nationally agreed Allergy Action Plans for Children38) with correspondence to patients/parents explaining that it is their responsibility to share this with nurseries/schools may help. School-based interventions can improve health outcomes, such as asthma,39 and future integration of care between education authorities, schools and healthcare providers with key roles for school nurses is critically important.
Our findings on information needs are not unexpected. Previous qualitative studies involving patients/parents with allergies identified common themes, such as medication use, effects/side effects, trigger factors, avoidance and causes.40 ,41 Future training of HCPs delivering services to patients with allergies should include these themes, and better signposting to further information needs to occur.
Conclusion
This new PREM for children with allergic disease allows children and their parents/carers to feed back their experiences of allergy care. Feedback will allow targeted education and improvement to existing services. Service users as well as service providers must be encouraged to use PREMs to inform their own practice and to shape future allergy services when negotiating with commissioners.
Acknowledgments
The authors wish to thank all participating children and their parents, as well as their multidisciplinary teams at Academic Paediatric Allergy Imperial College, London, and the Department of Paediatric Allergy, St Mary's Hospital, Imperial College Healthcare NHS Trust, London.
References
Footnotes
Contributors CG contributed to data collection/interpretation and wrote the manuscript. RG and COK contributed to data collection and had input in the manuscript. TR contributed to the development of the PREMs, to data collection/interpretation and to the manuscript. SS analysed the data and provided statistical support. AT and BH led the development stage using Picker Methodology, facilitated focus groups and cognitive interviews and contributed to the manuscript. JOW conceived and led the RCPCH allergy care pathways project and contributed to the development of the PREMs and to the manuscript.
Funding The work was funded by the Royal College of Paediatrics and Child Health as part of the Allergy Care Pathways Project and also by additional funding by NIHR CLAHRC North West London. This article presents independent research commissioned by the National Institute for Health Research (NIHR) under the Collaborations for Leadership in Applied Health Research and Care (CLAHRC) programme for North West London. The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.
Competing interests None declared.
Ethics approval Imperial College Healthcare NHS Trust.
Provenance and peer review Not commissioned; externally peer reviewed.