Use of the CPD-REACTION Questionnaire to Evaluate Continuing Professional Development Activities for Health Professionals: Systematic Review

Background Continuing professional development (CPD) is essential for physicians to maintain and enhance their knowledge, competence, skills, and performance. Web-based CPD plays an essential role. However, validated theory–informed measures of their impact are lacking. The CPD-REACTION questionnaire is a validated theory–informed tool that evaluates the impact of CPD activities on clinicians’ behavioral intentions. Objective We aimed to review the use of the CPD-REACTION questionnaire, which measures the impact of CPD activities on health professionals’ intentions to change clinical behavior. We examined CPD activity characteristics, ranges of intention, mean scores, score distributions, and psychometric properties. Methods We conducted a systematic review informed by the Cochrane review methodology. We searched 8 databases from January 1, 2014, to April 20, 2021. Gray literature was identified using Google Scholar and Research Gate. Eligibility criteria included all health care professionals, any study design, and participants’ completion of the CPD-REACTION questionnaire either before, after, or before and after a CPD activity. Study selection, data extraction, and study quality evaluation were independently performed by 2 reviewers. We extracted data on characteristics of studies, the CPD activity (eg, targeted clinical behavior and format), and CPD-REACTION use. We used the Mixed Methods Appraisal Tool to evaluate the methodological quality of the studies. Data extracted were analyzed using descriptive statistics and the Student t test (2-tailed) for bivariate analysis. The results are presented as a narrative synthesis reported according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. Results Overall, 65 citations were eligible and referred to 52 primary studies. The number of primary studies reporting the use of CPD-REACTION has increased continuously since 2014 from 1 to 16 publications per year (2021). It is available in English, French, Spanish, and Dutch. Most of the studies were conducted in Canada (30/52, 58%). Furthermore, 40 different clinical behaviors were identified. The most common CPD format was e-learning (34/52, 65%). The original version of the CPD-REACTION questionnaire was used in 31 of 52 studies, and an adapted version in 18 of 52 studies. In addition, 31% (16/52) of the studies measured both the pre- and postintervention scores. In 22 studies, CPD providers were university-based. Most studies targeted interprofessional groups of health professionals (31/52, 60%). Conclusions The use of CPD-REACTION has increased rapidly and across a wide range of clinical behaviors and formats, including a web-based format. Further research should investigate the most effective way to adapt the CPD-REACTION questionnaire to a variety of clinical behaviors and contexts. Trial Registration PROSPERO CRD42018116492; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=116492


Introduction
Continuing professional development (CPD) encompasses the multiple educational and developmental activities that health care professionals undertake to maintain and enhance their knowledge, skills, performance, and relationships in the provision of health care. The ultimate goal of CPD is to enhance the quality and safety of patient care and enhance both patient experience and health outcomes [1]. In recent years, web-based CPD has increased exponentially, and the recent COVID-19 pandemic has emphasized the need for more effective web-based CPD. Health professional behavior change (adoption or abandonment of a practice) is a long and complex process [2]. The Kirkpatrick model conceptualizes a framework for CPD assessment that measures four distinct outcome levels: satisfaction; knowledge, skills, or attitudes; transfer of learning to practice (ie, behavior); and organizational outcomes such as productivity and quality [3].
The lack of validated instruments informed by behavior change theories for assessing CPD outcomes has slowed the advancement of the CPD knowledge base [4]. In 2011, a consortium of CPD providers from the Province of Quebec, Canada, developed a tool to assess Kirkpatrick level 3 outcomes (transfer of learning to practice) based on an integrated model explaining behavior change among health professionals [5,6]. This model posits that intention is a strong predictor of behavior, and that behavioral intention, in turn, is influenced by beliefs about capabilities, beliefs about consequences, moral norms, and social influences [5]. The resulting tool, the CPD-REACTION questionnaire, is a comprehensive, theory-based, validated instrument for assessing the impact of accredited CPD activities on clinical behavioral intention [7,8]. During the past 10 years, it has been used in regular evaluations of the effects of CPD activities on behavior change by major CPD providers such as the Federation of Medical Specialists of Quebec (Fédération des Médecins Spécialistes du Québec) and to assess training for a wide variety of other health care professionals [9][10][11][12].
However, the current range of CPD-REACTION use remains unknown. Moreover, the clinical topics of CPD activities evaluated using the tool, the types of clinical behaviors sought, how often it has been used to evaluate web-based CPD, what kind of health care professionals are targeted by such CPD activities, and how the results shown by CPD-REACTION in terms of behavior change intentions are used, are also unknown.
Although tool validity has been demonstrated in the Canadian context [8], other evidence on its cross-cultural validity and psychometric properties is still lacking. Therefore, we aimed to systematically review studies that have used the CPD-REACTION questionnaire.
Our research questions were as follows: (1) What are the characteristics of CPD activities in studies using CDP-REACTION? (2) What are the ranges of behavioral change intentions, mean scores, and distribution of scores across all studies that used CPD-REACTION? (3) What are the psychometric properties of CPD-REACTION?

Ethics Approval and Consent to Participate
As this research was based on published studies, ethics approval was not required for this systematic review. The protocol was registered in the PROSPERO (International Prospective Register of Systematic Reviews) registry under the number CRD42018116492 on December 4, 2018. The main change to the protocol was the inclusion of references to studies reported in a language other than English and French.

Study Design
Informed by the Cochrane review methodology [13], we conducted a systematic review and followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 statement [14].

Eligibility Criteria
Informed by the PICOS (Population, Intervention, Comparator, Outcomes, Study design) model [15], the inclusion criteria were as follows: (1) Population-the target population considered for this review included all individuals working in health fields who completed an original, translated, or adapted version of the CPD-REACTION questionnaire before, after, or before and after an activity. There was no age restriction or restriction of health care professions (eg, physician, nurse, or any other health professional). They could be working in the public or private sector, in the process of training, or have already graduated; (2) Intervention-not specified; (3) Comparator-not specified; (4) Outcomes-The original, adapted, or translated version of the 12-item CPD-REACTION was used to assess the intention to change either a clinical practice or a health behavior. On the basis of the Godin integrated model, this tool is a questionnaire composed of 12 items that measure behavioral intention and some of its predictors; that is, beliefs about capabilities, beliefs about consequences, moral norms, and social influences [5,7]. The 5 constructs of the CPD-REACTION questionnaire have been validated, with Cronbach coefficients for the constructs varying from 0.77 to 0.85 [7,8]; (5) Study Design-Any study design was considered: randomized clinical trials (individual, group, or cluster, including stepped-wedge), before-and-after studies, translation studies, ecology studies, qualitative studies, or any mixed-study design (if they included the use of CPD-REACTION). Only primary studies were considered for inclusion in this systematic review. Therefore, we did not include any systematic reviews. These articles could be reported in any language.

Information Sources
The literature search was performed using eight databases: Embase, MEDLINE/PubMed, Web of Science, ERIC-EBSCO, PsycINFO-ovid, CINAHL, Social Sciences Full Text-EBSCO, and Academic Search Premier EBSCO. A temporal filter was applied from January 1, 2014, to April 20, 2021, because CPD-REACTION was published in 2014 [7]. We also performed a forward citation search using Google Scholar and Research Gate to identify studies citing the 3 main studies on the development and validation of CPD-REACTION (Multimedia Appendix 1).

Search Strategy
The first phase of developing the search strategy was carried out on PubMed and reviewed by the authors to ensure that the concepts covered all research questions. This strategy was then translated into expressions that were adapted to each database. A documentary research expert revised the search strategy and the final version was based on three key concepts: "continuing education," "CPD-REACTION questionnaire," and "questionnaires." These key concepts were searched using a combination of controlled vocabulary (MeSH [Medical Subject Headings] terms) and free-text search queries (Multimedia Appendix 1).

Selection Process
Duplicates were identified using EndNote ×9 [16] and manual checking. First, reviewers (GA-V, FBK, LB, and LS) performed an independent selection based on the title and abstract. Second, all relevant references were considered for selection by full text (GA-V, FBK, LL, LB, and LS). An internet-based system, Covidence [17], was used to complete this step. The 2 reviewers then discussed and resolved any disagreement to obtain a consensus on study selection according to the eligibility criteria and, if necessary, consulted a third author (KVP). The reasons for exclusion of articles were documented.

Data Extraction
A coding guide and corresponding extraction grid were developed and tested by the reviewers. The reviewers (GA-V, FBK, LL, LB, and LS) individually extracted data from the included studies. The reviewers discussed and resolved any disagreement.
Qualitative and quantitative data were extracted. The main groups of variables were (1) study characteristics, including author names, study design, study objectives, country, and type of CPD activity; (2) characteristics of the study participants, such as profession, setting (eg, hospital or university), average age, sex, study population (eg, single profession, mixed professions, and patients included); (3) CPD activity characteristics, such as country in which it was used, health field, duration of CPD activity, when tool was used (eg, pre-or post-CPD activity), format of CPD activity (eg, web-based), title of CPD activity, clinical behavior change targeted; (4) CPD-REACTION version used (original or adapted), adaptations to the questionnaire, eg, translations; (5) score values (mean, median, SD, minimum, and maximum) for all constructs measured, that is, behavioral intention, beliefs about capabilities, social influence, moral norm, and beliefs about consequences; (6) psychometric properties (Cronbach α, κ, or Cohen d).

Methodological Quality Assessment of Individual Studies
Two examiners (GA-V and FBK) assessed the quality of each identified study using the Mixed Methods Appraisal Tool (MMAT), a validated tool for evaluating the quality of qualitative, quantitative, and mixed methods studies [18]. For each type of study design, 5 criteria were evaluated and each was rated "yes," "can't tell," or "no." The tool guideline discourages the calculation of an overall score, instead suggesting presenting detailed ratings for each criterion [18] (Multimedia Appendix 2).

Data Synthesis
Given the large variety of behavior changes targeted by studies (clinical practice behaviors and others) and the methodological and statistical heterogeneity of studies, we performed a narrative synthesis using descriptive statistics. For the CPD-REACTION score values, we did not calculate the average scores for the construct if CPD-REACTION did not evaluate the same behavior. Instead, we summarized the construct scores based on the timing of the evaluation, that is, if it was a pre-post, only pre-evaluation, or only postevaluation. Descriptive statistics were computed using STATA (version 11; StataCorp). To summarize the target behaviors of the included studies, we performed a thematic analysis. After the analysis, we organized and summarized the main behaviors based on the emerging themes, namely, "shared decision-making," "decision aids or toolkit," and "others."

Risk of Bias in Studies
Although none of the studies fully met all MMAT criteria, none were rated "no" for any criteria (Multimedia Appendix 2). In the 4 quantitative randomized trials, only the criterion "randomization appropriately performed" was met by all 4 studies [10,50,54,63], and in all 17 mixed method studies, only the criterion "adequate rationale for using a mixed methods design" was met. In all the design groups, all the criteria not rated "yes" were rated "not sure."

Principal Findings
We found 61 publications of 52 unique studies that reported the use of the CPD-REACTION questionnaire to assess changes in behavioral intention among health professionals. Although the tool is aged <10 years (2014), we observed the most rapid increase in its use in the past 3 years, mostly in Canada, where it was developed. However, its use has spread to many other countries, including lower-and middle-income countries, and it is found in numerous languages (our finding of only 4 is an underrepresentation, as the team that produced the tool has agreed to translations into 8 languages) [75]. Since its inception, CPD-REACTION has been used by close to 5000 participants to target 39 clinical behaviors. The participants included 8 types of health professionals, with physicians and nurses being the most reported. Two-thirds of the studies included interprofessional clinical teams, including one in which 10 managers or decision-makers were CPD activity participants. The tool appeared to be mostly used for evaluating e-learning (n=34). In many cases, users adapted the questionnaire, such as using only certain construct scales or adding or dropping some items. The psychometric properties of CPD-REACTION reported in included studies showed that Cronbach α scores were very good, ranging from 0.62-0.91. However, few studies were designed to assess changes in intention (ie, scoring both pre-and postactivity), thus limiting the evidence regarding the responsiveness of the tool. Regarding behavioral intention to change, the mean difference of intention score was 0.54 SD 0.13 in the pre-post studies and the distribution of scores across all studies using CPD-REACTION ranged from 1.8-7. Although none of the studies fully met all MMAT criteria, none were rated "no" for any criteria. In all the design groups, all the criteria not rated "yes" were rated "not sure."

Significance and Comparison With Prior Work
First, the rapid adoption of CPD-REACTION across time, countries, and languages suggests that this instrument addresses the needs of CPD developers and that they seek not only validated assessment tools but also those that are informed by behavior change theories. Recent literature on this topic tends to suggest an increasing penetration of behavior change theories in the CPD developer community [2,76,77]. The use of behavior change theory has been frequently linked to effectiveness in systematic reviews of behavioral change interventions [76,77]. More recently, strategies have also focused not only on adopting new behaviors but also on abandoning low-value or harmful behaviors. However, few behavioral theories distinguish between behavior adoption and abandonment, including the theories on which CPD-REACTION is based [78]. Future research should distinguish between the two and develop theories that support both types of behavior change [79,80].
Second, physicians and nurses were the most represented health professionals. Most groups of participants engaging in CPD activities were interprofessional clinical teams, and 1 in 10 studies included managers or decision-makers among participants. This suggests that CPD designers are increasingly creating multidisciplinary training experiences to be shared with other stakeholders and professionals to enhance the relevance and impact of CPD [2,81]. Previous research has highlighted that including peer groups seems to be an effective approach to enhancing CPD activities and moving forward with professional practice change [82]. Future research should determine the effects of interprofessional participant groups or peer groups on CPD effectiveness.
Third, studies using an adapted version of CPD-REACTION reported Cronbach α ranging from .62 to .91, indicating that modified instruments perform well in terms of their psychometric properties. Other studies have reported psychometric values mentioned in the original version of CPD-REACTION. We observed that overall, the behavioral change intention scores reported ranged from 2.9 to 7. In pre-post studies, the mean difference in intention scores was 0.54 SD 0.13, and the distribution of scores across all constructs ranged from 1.8 to 7. Lower scores were observed when CPD-REACTION was used either only preactivity or only postactivity. Dissemination of the user manual will aid in the use of the tool to its best advantage. A lower score could also be because of the CPD topic being more controversial and thus less likely to be implementable. Overall, the adapted versions of CPD-REACTION reported Cronbach α values, indicating that the questionnaire had good internal consistency reliability. Furthermore, our results suggest that CPD-REACTION is adaptable to digital platforms, as two-thirds of the activities were web-based.
Fourth, using CPD-REACTION to measure construct scores, both pre-and post-CPD activity, is a helpful demonstration of the effect of CPD activities on behavioral intention and explanatory constructs. However, measuring learning outcomes for levels 3 and 4 of the Kirkpatrick model remains challenging. CPD-REACTION uses intention as a measure of behavioral intention; however, other measurement strategies are needed to directly measure behavior change. Although other outcomes such as "satisfaction of participants" were reported, the studies did not correlate these with the CPD-REACTION measures. In some studies, participants were contacted after 3 months or more to self-assess their behavior change [8]. The purpose of CPD-REACTION was not to measure its effects on patient outcomes, which is another important outcome of CPD. CPD-REACTION could be followed up by participant surveys to assess the longer-term impacts of participants' behaviors on their practices or institutions and should use patient-reported measures. Some studies suggest that CPD programs should compare self-assessments, such as CPD-REACTION, with continuous formal participant multisource assessment by peers [73].

Limitations
Our systematic review used diverse strategies to find studies that had used CPD-REACTION. However, we relied on the published results and did not contact the authors of the included studies. Thus, it is possible that we may have missed studies that were not published as well as items of interest in those we included. Owing to the large number of included studies, we had to organize the information into broad categories to increase the interpretability of the data.

Conclusions
The CPD-REACTION questionnaire is a simple, relevant, and easy-to-use tool for assessing the effectiveness of CPD activities on health professionals' behavioral intention and, as we have observed, to identify barriers and facilitators of behavior change. This tool has been used to evaluate CPD activities in a wide range of clinical topics and behaviors. However, most users do not measure intention both before and after the activity. Dissemination of a user manual will aid in the use of the tool to its best advantage. Further research should investigate the most effective way to adapt the CPD-REACTION questionnaire to various uses and contexts.