The effect of audit and feedback on immunization delivery: A systematic review

doi:10.1016/S0749-3797(00)00126-4

American Journal of Preventive Medicine

Volume 18, Issue 4, May 2000, Pages 343-350

https://doi.org/10.1016/S0749-3797(00)00126-4 Get rights and content

Abstract

Objective: To assess the effective of audit and feedback (A&F) on immunization delivery by health care professionals.

Design: Systematic review of published literature.

Main Outcome Measures: Changes in immunization rates.

Methods: We searched Medline between 1966 and 1997. We obtained additional studies from back-searching reference lists and the files of study collaborators. We included studies that were written in English, that included audit and feedback in at least one arm of the study, that studied universally recommended childhood or adult vaccines, and that provided immunization coverage data. Two reviewers read studies independently and abstracted using a validated checklist. Study quality was assessed using criteria standardized by the Cochrane Collaboration. Differences between reviewers were resolved by consensus.

Results: The search process resulted in 60 citations; 44 were fully reviewed and 15 met eligibility criteria. Five were randomized trials. Twelve of the fifteen studies found that A&F, alone or in combination with other interventions, were associated with improvements in immunization rates. The magnitude of the effect varied from −17% to +49% change. Study design heterogeneity precluded statistical pooling of study results.

Conclusions: The evidence available from published studies suggests that A&F alone may be an effective strategy for improving immunization rates. The number of well-conducted studies is small, and the effect is variable. Additional well-designed studies are needed to identify the independent effects of A&F, optimal format and frequency of A&F, and to examine its long-term effect on provider immunization practices and costs.

Introduction

Immunization is a cost effective and widely accepted means of preventing disease. However, the percentage of 2-year-old children up-to-date for immunizations in the United States remains below the U.S. Public Health Service goal of 90%.1 Adult immunizations, such as pneumococcal immunization for the elderly, also fall below targeted rates.2 Audit and feedback (A&F) is one of the numerous strategies devised to improve immunization rates and other preventive care services. Audit in medical practice usually refers to the collection of data regarding clinical performance by reviewing medical charts, laboratory orders, or billing records; whereas feedback refers to the presentation of such data to the relevant clinician, with or without recommendations for practice.3 After the implementation of routine A&F to providers in public health clinics in the state of Georgia, the percentage of 2-year-olds up-to-date for immunizations increased from 40% to 89% over 8 years.4 Although the intervention in Georgia was multifactorial, the apparent positive effect of A&F led the U.S. Congress in 1995 to require that all states receiving federal money for immunizations perform routine A&F of immunization rates in public clinics.5 Similarly, the Advisory Committee on Immunization Practices has recommended that private providers, as well as those in public settings, implement routine measurement of immunization coverage.6 These recommendations have been promulgated without a preponderance of evidence to support the effectiveness of these measures.

A review by Buntinx7 of the effect of A&F on a variety of preventive services found that feedback does increase adherence to clinical practice guidelines, particularly when clinicians are involved in the development of those guidelines. A review by Thomson8 concluded that the effect of feedback on various clinical services is generally small and that current evidence regarding optimal frequency and format is too variable to support any recommendations for practice.

Perhaps the effect of audit and feedback differs according to the specific preventive service for which it is used. Because immunization delivery is generally well accepted and a priority for providers and patients or their parents, the effect of feedback on immunization rates may be greater than with other preventive services not as universally accepted. Other features unique to immunization delivery, such as the complex schedule of immunizations recommended for children, may modify the impact of A&F when compared with other preventive services. Audit and feedback addresses one barrier to immunization delivery, lack of provider information, by giving providers an objective estimate of the immunization coverage of their patients. Even if A&F has no immediate direct effect, it may motivate providers to examine other potential barriers to immunization delivery in their practice settings (e.g., high rates of missed opportunities) and to implement alternative approaches to address them.

In the absence of effective electronic clinical information systems, collecting, analyzing, and reporting audit and feedback data is time-consuming and expensive. If providers nationwide are expected to adopt A&F as a strategy to potentially improve immunization rates, this approach needs to be grounded in science. We performed this systematic review of the literature to summarize the effectiveness of A&F on immunization delivery and to identify implications for future research.

Section snippets

Search strategy for identification of studies

We performed a Medline search to identify articles published between 1966 and 1998, using the search terms: “quality of health care” (mh), “physician practice patterns” (mh), “audit” (tw), “reaudit” (tw), “assessment” (tw), “outcome assessment” (health care) (mh), “feedback” (mh), “feedback” (tw), “feed-back” (tw), “fed back” (tw), “immunization” (mh), “immunize” (tw), “preventive health services” (mh), “vaccines” (mh), “immunization programs” (mh), “vaccine” (tw), “vaccination” (mh),

Results of literature search

We identified 37 studies by the literature search. An additional 23 were found through back-searches. Fifteen studies met eligibility criteria and were included in the final review. Reviewers reached 100% agreement on study inclusion. The majority of articles were excluded because of study design issues (e.g., ecologic studies), failure to present interpretable data, or because they were not reports of individual studies (e.g., reviews). When we applied the more stringent Cochrane Collaboration

Discussion

We identified numerous studies that examined the effect of A&F on immunization delivery. Twelve of the fifteen that met eligibility criteria found that A&F, alone or in combination with other interventions, were associated with improvements in immunization rates. However, only five of these studies were RCTs. The remainder were a mix of before-and-after and ITS studies. Following the 1989–1991 measles epidemic, significant national, state, and local attention was given to increasing childhood

Acknowledgements

Funding for this study was provided via a cooperative agreement between the Ambulatory Pediatric Association and the Centers for Disease Control and Prevention, Atlanta, Georgia.

References (26)

T.S Carey et al.
Development of a model quality-of-care assessment program for adult preventive care in rural medical practices
QRB Qal Rev Bull
(1991)
C.H Chodroff
Cancer screening and immunization quality assurance using a personal computer
QRB Qual Rev Bull
(1990)
National, state, and urban area immunization coverage levels among children aged 19-35 months—United States, January-December 1995
MMWR Morb Mortal Wkly Rep
(1998)
Influenza and pneumococcal vaccination levels among adults aged greater than or equal to 65 years—United States, 1997
MMWR Morb Mortal Wkly Rep
(1998)
A.D Oxman et al.
No magic bulletsa systematic review of 102 trials of interventions to improve professional practice
Can Med Assoc J
(1995)
E.F Dini et al.
Information as interventionhow Georgia used vaccination coverage data to double public sector vaccination coverage in seven years
J Pub Health Manage Pract
(1996)
Evaluation of vaccination strategies in public clinics—Georgia, 1985–1993
MMWR Morb Mortal Wkly Rep
(1995)
Recommendations of the Advisory Committee on Immunization Practicesprogrammatic strategies to increase vaccination rates—assessment and feedback of provider-based vaccination coverage information
MMWR Morb Mortal Wkly Rep
(1996)
F Buntinx et al.
Influencing diagnostic and preventive performance in ambulatory care by feedback and reminders. A review
Fam Pract
(1993)
M.A Thomson et al.
Audit and feedback to improve health care professional practice and health care outcomes

Reviews for the Cochrane Collaboration on Effective Professional Practice (CCEPP)

D Moher et al.

Guide for reading and interpreting systematic reviews III. How did the authors synthesize the data and make their conclusions?

Arch Pediatr Adolesc Med

(1998)

A.L Hillman

Pediatric preventive care incentives in a Medicaid HMO

(1996)

Cited by (57)

Establishing and Maintaining a Vaccine-Positive Practice Culture
2020, Primary Care - Clinics in Office Practice
A review of the key factors to improve adult immunization coverage rates: What can the clinician do?
2018, Vaccine
Adult immunization coverage rates remain low in the United States and internationally, despite obvious benefits to vaccinating and maintaining a well-vaccinated adult population. Broad policy changes are required to identify and address gaps in financing, in immunization infrastructure, and patient and provider awareness and knowledge to improve the protection of our adult and aging population from vaccine-preventable diseases. There is good evidence that efforts are now underway both within the United States and across the world to advance these policy changes. There are successful interventions that have been demonstrated to improve rates in the pediatric population that must be translated into the adult patient population to meet the critical gaps that remain at the interface of the delivery of vaccinations to adults. Improvements in overall policy will only increase adult immunization coverage rates if interventions are adapted and implemented for adult patients. Often, these same interventions will be applicable to adolescent patients as well. These interventions have been reviewed by the United States Preventive Services Task Force and recommended interventions fall into three categories: (1) Enhancing Patient Access to Vaccination; (2) Improving community/patient demand; and (3) Provider- and healthcare system-directed interventions. Specific interventions that have been demonstrated successful for the adult population include interventions such as reducing patient out-of-pocket costs for vaccinations, patient or family incentive rewards, and implementation of quality measures and quality improvement interventions. Addressing the poor performance in adult immunizations requires approaches predicated on not repeating previous efforts and will require innovative thinking to integrate multiple interventions that have been successful separately, into a holistic approach to support and automate immunization assessment, recommendation, and administration. This can then lead to increased valuation of adult and adolescent immunizations within the priorities of a healthcare system, and improvements in clinic efficiency within a practice.
Seasonal influenza vaccination at school: A randomized controlled trial
2014, American Journal of Preventive Medicine
Citation Excerpt :
Despite obstacles presented by needing consent and insurance information prior to vaccination, SLV-I schools had 6% to 16% higher vaccination coverage than control schools, and the odds of vaccination were 50%–60% higher for students in SLV-I than control schools. Although this intervention did not lead to achievement of goal levels of coverage or an increase in school age vaccination sufficient to have, by itself, a major impact on the overall community level of influenza disease, this trial did produce an impact that exceeded most interventions recommended to increase childhood vaccination coverage, for example, sending immunization reminder messages or decreasing missed opportunities to vaccinate during visits.42–46 Because of the importance of influenza vaccination to the health of the individual and the community, we believe the improvement demonstrated in this trial is considerable.
Influenza vaccination coverage for U.S. school-aged children is below the 80% national goal. Primary care practices may not have the capacity to vaccinate all children during influenza vaccination season. No real-world models of school-located seasonal influenza (SLV-I) programs have been tested.
Determine the feasibility, sustainability, and impact of an SLV-I program providing influenza vaccination to elementary school children during the school day.
In this pragmatic randomized controlled trial of SLV-I during two vaccination seasons, schools were randomly assigned to SLV-I versus standard of care. Seasonal influenza vaccine receipt, as recorded in the state immunization information system (IIS), was measured.
Intervention and control schools were located in a single western New York county. Participation (intervention or control) included the sole urban school district and suburban districts (five in Year 1, four in Year 2).
After gathering parental consent and insurance information, live attenuated and inactivated seasonal influenza vaccines were offered in elementary schools during the school day.
Data on receipt of ≥1 seasonal influenza vaccination in Year 1 (2009–2010) and Year 2 (2010–2011) were collected on all student grades K through 5 at intervention and control schools from the IIS in the Spring of 2010 and 2011, respectively. Additionally, coverage achieved through SLV-I was compared to coverage of children vaccinated elsewhere. Preliminary data analysis for Year 1 occurred in Spring 2010; final quantitative analysis for both years was completed in late Fall 2012.
Results are shown for 2009–2010 and 2010–2011, respectively: Children enrolled in suburban SLV-I versus control schools had vaccination coverage of 47% vs 36%, and 52% vs 36% (p<0.0001 both years). In urban areas, coverage was 36% vs 26%, and 31% vs 25% (p<0.001 both years). On multilevel logistic analysis with three nested levels (student, school, school district) during both vaccination seasons, children were more likely to be vaccinated in SLV-I versus control schools; ORs were 1.6 (95% CI=1.4, 1.9; p<0.001) and 1.5 (95% CI=1.3, 1.8; p<0.001).
Delivering influenza vaccine during school is a promising approach to improving pediatric influenza vaccination coverage.
ClinicalTrials.govNCT01224301
Strategies to improve vaccination uptake in Australia, a systematic review of types and effectiveness
2012, Australian and New Zealand Journal of Public Health
To identify and critique interventions to improve vaccination uptake in Australia.
Peer‐reviewed and grey literature from 1997 to May 2011 was searched to identify evaluations of one or more interventions to improve vaccination uptake among any target group in Australia. Studies were categorised by intervention type and target group. Recommended tools for assessing quality in public health interventions were used in the methodological critique of included studies.
Forty‐nine studies met the inclusion criteria, two‐thirds of which were published in peer‐reviewed journals. Evidence for strategies that increase community demand for vaccination was most common. Multi‐component strategies, patient and provider reminders, plans for catch‐up vaccination and accelerated schedules were identified as most effective. There was a lack of evidence for strategies to improve coverage in Aboriginal and Torres Strait Islander peoples, behaviourally at‐risk groups and pregnant women. Major limitations of identified studies were the lack of baseline coverage for comparison, limited use of controlled designs and measurement biases.
The evidence, while limited, suggests that the most effective strategies are those which increase community demand for and enhance access to vaccines. Strategies to increase vaccination uptake are infrequently and often inadequately evaluated, despite the need for evidence to support their use.
The results of this review, used in conjunction with international evidence, can guide those desiring to improve the performance of vaccination programs and suggest priorities for future evaluation of strategies to improve vaccination uptake.
Adolescent Vaccination: Physician Challenges and Solutions
2010, Journal of Adolescent Health
Citation Excerpt :
A Dutch study of 1,763 parents of 6–14-year-old children found that parents who thought they could intervene to prevent the disease (internal mode of control) were more likely to consider the side effects of meningococcal C vaccine as being serious compared with other parents [41]. A Cochrane review of audit and feedback (A&F) of childhood and adult vaccines found that A&F was associated with improved immunization rates in 12 of 15 studies [42]. However, providers were admonished to balance potential benefits of A&F with its potential costs [42].
As additional vaccines for adolescents come to market, physicians are confronted with multiple barriers to providing immunizations, including vaccine costs, concerns about safety, attitudes and/or knowledge, and obtaining adolescent consent. Immunization rates in adolescents could be improved by increasing providers' knowledge about recommended vaccines, providing information about vaccine safety to parents and patients, and taking care not to miss any vaccination opportunities.
Translational Research and Pediatrics
2009, Academic Pediatrics
Citation Excerpt :
Considering the example of pediatric immunizations, T2 studies include phase 3 vaccine trials such as the recent studies of rotavirus vaccine efficacy84,85 that led to new immunization recommendation.86 Other immunization examples include systematic reviews of efficacy studies of patient reminder/recall87 and provider prompts.88 For pediatric asthma, examples include phase 3 trials of asthma medications, clinical trials of office-based prompting to improve the delivery of preventive asthma care in controlled settings, and randomized trials of school-based delivery of preventive asthma medications to improve asthma care.89–93

View all citing articles on Scopus

View full text

Review and Special ArticlesThe effect of audit and feedback on immunization delivery: A systematic review

Abstract

Introduction

Section snippets

Search strategy for identification of studies

Results of literature search

Discussion

Acknowledgements

QRB Qal Rev Bull

QRB Qual Rev Bull

National, state, and urban area immunization coverage levels among children aged 19-35 months—United States, January-December 1995

MMWR Morb Mortal Wkly Rep

Influenza and pneumococcal vaccination levels among adults aged greater than or equal to 65 years—United States, 1997

MMWR Morb Mortal Wkly Rep

No magic bulletsa systematic review of 102 trials of interventions to improve professional practice

Can Med Assoc J

Information as interventionhow Georgia used vaccination coverage data to double public sector vaccination coverage in seven years

J Pub Health Manage Pract

Evaluation of vaccination strategies in public clinics—Georgia, 1985–1993

MMWR Morb Mortal Wkly Rep

Recommendations of the Advisory Committee on Immunization Practicesprogrammatic strategies to increase vaccination rates—assessment and feedback of provider-based vaccination coverage information

MMWR Morb Mortal Wkly Rep

Influencing diagnostic and preventive performance in ambulatory care by feedback and reminders. A review

Fam Pract

Audit and feedback to improve health care professional practice and health care outcomes

Reviews for the Cochrane Collaboration on Effective Professional Practice (CCEPP)

Guide for reading and interpreting systematic reviews III. How did the authors synthesize the data and make their conclusions?

Arch Pediatr Adolesc Med

Pediatric preventive care incentives in a Medicaid HMO

Review and Special Articles
The effect of audit and feedback on immunization delivery: A systematic review