Skip to main content

Problem-based or lecture-based learning, old topic in the new field: a meta-analysis on the effects of PBL teaching method in Chinese standardized residency training

Abstract

Background

Standardized residency training (SRT) is crucial for graduate medical education and the training of high-quality doctors. Nevertheless, China started SRT nationwide only in the recent decade. During these years, researchers have been searching for suitable teaching methods to improve the abilities of residents. Although the problem-based learning (PBL) teaching mode has been applied in undergraduate teaching for many years, the teaching effect of PBL has not been unified in Chinese SRT according to the core competences of the residents.

Methods

Studies that compared the teaching effect of PBL and lecture-based learning (LBL) on SRT in China from January 2010 to April 2020 in the Chinese databases, such as China National Knowledge Infrastructure (CNKI), WanFang, WeiPu, Chinese BioMedical Literature (CBM), and English-language online databases, such as PubMed, Embase, and Cochrane Library were systematically reviewed. Data were analyzed using the Stata version 12.0 software.

Results

A total of 75 articles (76 studies) were included in this meta-analysis. Compared with LBL group, PBL-based methods are more effective in the mastery of medical theory knowledge (WMD = 7.14, 95% CI: 5.93–8.34), operational skills (WMD = 6.54, 95% CI: 4.55–8.53), analysis and diagnosis of cases (WMD = 8.52, 95% CI: 7.50–9.53), and overall capacity (WMD = 8.70, 95% CI: 6.87–10.53), but showed no advantage on operational skills in diagnostic imaging (WMD = 1.30, 95% CI: -0.11–2.71). The questionnaire surveys analyzed in this meta-analysis indicated the positive effects of PBL on the mastery of theoretical knowledge, clinical diagnostic thinking, teamwork ability, ability to analyze and solve problems, ability to consult documents, learning interest and learning efficiency, but that there were no advantages in improving self-directed learning ability, communication ability and hands-on ability. The questionnaire result analyzed in this meta-analysis also showed the residents’ satisfaction with PBL-based strategies.

Conclusions

Taken together, the current meta-analysis provides a systematic and comprehensive analysis on PBL teaching mode in Chinese SRT and outlines a path for further research on the detailed design of suitable teaching methods for different specialties and abilities.

Peer Review reports

Introduction

Standardized residency training (SRT) occupies a very important role in connecting the basic education of medical colleges and continuing medical education, and it is the key tool and pathway to train qualified clinicians for improving the overall medical level. However, the unified nationwide reform of medical education about SRT in mainland China did not begin until 2013, termed as 5 + 3 model, encompassing 5 years of undergraduate medical studies (leading to a Bachelor degree) and 3 years of SRT in one of the 36 specialties [1, 2]. Compared to the developed countries, wherein the SRT has been gradually maturing after a hundred years, this program is still in its infancy [3, 4].

With increasing focus on cultivation of competencies which is the critical problem of SRT, selecting a suitable type of teaching method is needed urgently [5, 6]. Problem-based learning (PBL), of which the training objectives are consistent with those of resident trainees, has been carried out in some residency training bases in recent years [7, 8]. However, whether PBL is better than lecture-based learning (LBL) which is the primary teaching method in the Chinese medical education system, there still is no uniform conclusion [9].

A common limitation of previous studies on this topic is that they all include the research before the nationwide reform of SRT [10, 11]. Before performing SRT, majority of the medical students have been directly engaged in clinical work in hospitals at different levels, and it has a severe impact on the homogenization training of the residents without unified standards. The SRT with guidelines for each specialty rotation about required time, purpose, requirements and assessments, would ensure that medical school graduates receive standardized and institutionalized training in the certified training institutions [12]. Therefore, it is of great importance to analyze the effects of PBL vs. LBL teaching method under the unified background of SRT reform.

Herein, we present a meta-analysis for the first time which only includes the studies performed under the nationwide reform of SRT. In addition, this meta-analysis includes the mastery of medical theory knowledge, operational skills, and analysis and diagnosis of the cases, while subgroup analyses based on the teaching methods and department types were also carried out. Moreover, questionnaire surveys (QS) about theoretical knowledge mastery, clinical diagnostic thinking, teamwork ability, ability to analyze and solve problems, communication ability, learning interest, self-directed learning ability, hands-on ability, ability to consult documents, learning efficiency, and satisfaction with teaching were systematically analyzed. Together, this accurate and comprehensive analysis would provide a scientific basis for the selection and application of teaching methods in Chinese SRT in the future.

Methods

Literature search

We searched China National Knowledge Infrastructure (CNKI), WanFang (Chinese database), WeiPu (Chinese database), Chinese BioMedical Literature (CBM), and English-language online databases, such as PubMed, Embase, and Cochrane Library. The following terms or keywords were used: “problem-based learning” OR “PBL”) AND (“case-based learning” OR “CBL”) AND (“standardized residency training” OR “standardized training” OR “SRT” OR “resident” OR “5 + 3 model”. Next, the references of the review articles were scanned for additional eligible reports. The search was restricted from January 2010 to April 2020; no language restrictions were imposed.

Inclusion criteria

The studies were included according to the following four criteria: (a) Target population: residents in SRT in China; (b) Study design: randomized controlled trials; (c) Interventions: PBL or PBL + CBL served as the experimental group and LBL comprised the control group; (d) Outcome measurements (at least one of these): knowledge scores (KS), were used to assess how well the residents mastered the related theoretical knowledge; skill scores (SS), which were used to assess the operational skills, such as urethral catheterization in Urology and endotracheal intubation in Anesthesiology; practical skills (PS) assessments, including medical history collection, physical examination, making diagnosis and treatment plan, were used to assess the ability of solving practical clinical problems; total scores (TS), which included knowledge scores, skill scores and practical skills scores, were used to assess the overall abilities; QS, which were self-reported questionnaire surveys, were used to assess the residents' recognition of the relevant aspects of the different teaching methods. The results of SS, PS and TS were presented as scores out of 100 and for QS, in which each item was assessed by a yes or no, the results were presented as percentages.

Exclusion criteria

The exclusion criteria were as follows: (a) Comprising of subjects other than residents; (b) The studies were non-randomized and non-controlled; (c) Utilized interventions other than PBL or PBL + CBL; (d) The control group was not LBL or combined with other methods; (e) No comparison of baseline indicators between the two groups; (f) Studies with partial data duplication.

Data extraction

Data were independently extracted by two reviewers. Any disagreements about the eligibility were resolved by consensus. The following information was extracted for each included study: (a) the first author, (b) the year of publication, (c) the study type, (d) the sample size (intervention and control groups), (e) the specialty of the residents, (f) characteristics of the residents, (g) characteristics of the tutors, (h) the intervention methods, (i) year of residency training, (j) the duration of intervention, and (k) the outcome measures.

Quality assessment

The quality of each included study was assessed using the risk of bias table according to the Cochrane Collaboration by two reviewers independently [13]. Any disagreement was resolved by discussion to achieve a consensus. The following quality items were checked: (a) random sequence generation, (b) allocation concealment, (c) blinding of participants and personnel, (d) blinding of outcome assessment, (e) incomplete outcome data, (f) selective reporting, and (g) other sources of bias.

Statistical analysis

Data were analyzed using the Stata version 12.0 software. The effect sizes on scores were presented by weighted mean difference (WMD) and 95% confidence intervals (CIs), and those on questionnaires were presented by odds ratios (ORs) and 95% CIs. The chi-squared test-based Q-statistic and I2 statistic was used to estimate the heterogeneity (I2 ≤ 25%, low heterogeneity; 25% < I2 < 50%, moderate heterogeneity; I2 ≥ 50%, and high heterogeneity) [14]. A fixed-effects model was used to pool the results when heterogeneity was ≤ 50%, while a random-effects model was applied when heterogeneity was > 50% [15, 16]. Sensitive analysis was performed to investigate the influence of a single study on the overall pooled estimate by sequential deletion of each study. Subgroup analysis according to teaching methods and departments was conducted. The publication bias was evaluated by the Begg’s and Egger’s test [17, 18]. P < 0.05 indicated statistically significant publication bias.

Results

Search results

The flow diagram of the search strategy is illustrated in Fig. 1. A total of 1438 potentially relevant articles was identified, of which 347 duplicates were removed. At the screening stage, 813 articles were excluded after reading the titles and abstracts, among which 667 were not relevant to the topic, and 146 were reviews. According to the inclusion and exclusion criteria, 278 full-text articles were assessed for eligibility. Among these, 16 studies were non-randomized controlled trials, 55 did not include a control group, 23 did not use LBL in control group, 11 used other teaching methods in addition to LBL in the control group, 16 used other teaching methods in addition to the PBL or PBL + CBL in the intervention group, 34 did not compare the baseline indicators between the two groups, 4 had duplicate data, and 44 did not provide the required data. One article included two groups based on graduate and non-graduate students, so counted into two studies. Thus, a total of 75 articles (76 studies) were included in this meta-analysis [19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93] (Additional file 1: Table A1).

Fig. 1
figure 1

Schematic of the search strategy

Study characteristics

The characteristics of these 76 included studies, published in Chinese between 2010 and 2019, are listed in Table 1. The sample size of these studies was 10–108 residents in the intervention group and 12–107 in the control group, and the pooled sample size was 4597 (intervention group = 2323, control group = 2274). The included studies covered 26 specialties (23 Internal Medicine, 8 Neurology, 3 Emergency Medicine, 4 Pediatrics, 16 Surgery, 4 Anesthesiology, 3 Obstetrics and Gynecology, 3 Ophthalmology, 2 Dentistry, 4 Medical Sonography, 1 Psychiatry, 2 Radiation Oncology, 2 Radiology, and 1 Traditional Chinese Medicine). All the studies described the baseline information about residents and tutors in both groups. For residents, 68 studies were matched for age, 67 were matched for sex, 44 were matched for educational background (EB), 29 were matched for department entrance exam scores (DS), 3 were matched for clinical working hours, 1 was matched for PBL experience, 1 was matched for the duration of intervention, and 1 was matched for the year of residency training. For tutors, 30 studies were matched for professional titles (PT), 14 were matched for teaching experience (TE), 4 were matched for educational background (EB), 9 described that the tutors were the same in both groups, 5 described that tutors in both groups received PBL teaching training, and 6 described that tutors in both groups performed collective lesson preparation (CLP). In 47 studies, PBL was performed in the intervention groups, and in 29 studies, PBL + CBL was conducted in the intervention groups. The residents were in the first or second year of residency training, as mentioned in 7 studies, while 18 studies described the duration of intervention, 3 weeks–8 months.

Table 1 Characteristics of the included studies

There were 40, 22, 27 and 27 studies in KS, SS, PS and TS respectively. QS were used as the outcomes to evaluate several abilities among the theoretical knowledge mastery, clinical diagnostic thinking, teamwork ability, ability to analyze and solve problems, communication ability, learning interest, self-directed learning ability, hands-on ability, ability to consult documents, learning efficiency, and satisfaction with teaching in 37 studies.

Study quality

All the included studies were assessed for the risk of bias (Fig. 2). The studies were designed as randomized controlled trials, and the results were reported adequately. All studies were free of selective reporting and other biases. The allocation concealment and blinding were not stated in these studies.

Fig. 2
figure 2

Risk of bias assessment. A Risk of bias graph as percentages for all included studies; (B) Risk of bias summary for each included study

Effects of interventions (PBL and PBL + CBL) on KS

A total of 40 publications involving 2190 residents (intervention group = 1111 and LBL group = 1079) reported KS. Because a high heterogeneity was observed across these studies (I2 = 95.6%, P < 0.0001), the random-effects model was used. The pooled effect size showed a significant difference in KS (WMD = 7.14, 95% CI: 5.93–8.34, P < 0.0001) in favor of the intervention group compared to the LBL group (Fig. 3).

Fig. 3
figure 3

Forest plot for the effects of interventions (PBL and PBL + CBL) on knowledge scores compared to the LBL group

Effects of interventions (PBL and PBL + CBL) on SS

A total of 22 publications involving 1096 residents (intervention group = 547, LBL group = 549) reported SS. Because a high heterogeneity was observed across these studies (I2 = 96.2%, P < 0.0001), the random-effects model was used. The pooled effect size showed a significant difference in skill scores (WMD = 6.54, 95% CI: 4.55–8.53, P < 0.0001) in favor of the intervention group compared to the LBL group (Fig. 4).

Fig. 4
figure 4

Forest plot for the effects of interventions (PBL and PBL + CBL) on skill scores compared to the LBL group

Effects of interventions (PBL and PBL + CBL) on PS assessments

A total of 27 publications involving 1568 residents (intervention group = 787, LBL group = 781) reported PS assessments. Because a high heterogeneity was observed across all these studies (I2 = 89.8%, P < 0.00001), the random-effects model was used. The pooled effect size showed a significant difference in PS assessments (WMD = 8.52, 95% CI: 7.50–9.53, P < 0.0001) in favor of the intervention group compared to the LBL group (Fig. 5).

Fig. 5
figure 5

Forest plot for the effects of interventions (PBL and PBL + CBL) on practical skills assessments compared to the LBL group

Effects of interventions (PBL and PBL + CBL) on TS

A total of 27 publications involving 1542 residents (intervention group = 770, LBL group = 772) reported TS. Because a high heterogeneity was observed across these studies (I2 = 97%, P < 0.00001), the random-effects model was used. The pooled effect size showed a significant difference in the total score (WMD = 8.70, 95% CI: 6.87–10.53, P < 0.0001) in favor of the intervention group compared to the LBL group (Fig. 6).

Fig. 6
figure 6

Forest plot for the effects of interventions (PBL and PBL + CBL) on total scores compared to the LBL group

Subgroup analysis of outcome measurements

In order to explore the sources of heterogeneity, this study conducted a subgroup analysis of teaching methods and departments. The teaching methods were divided into PBL group vs. LBL group and PBL + CBL group vs. LBL group. The teaching departments were divided into non-surgery, surgery, and diagnostic. The data are shown in Table 2. Only diagnostic imaging did not display a statistical significance in SS, while the data from the other subgroups differed significantly compared to the LBL group. However, the heterogeneity was not reduced significantly.

Table 2 Subgroup analysis of outcome measurements

Effects of interventions assessed by QS

The questionnaire (Table 3.) showed that the intervention group is superior to the LBL group with respect to theoretical knowledge mastery, clinical diagnostic thinking, teamwork ability, ability to analyze and solve problems, ability to consult documents, learning interest, satisfaction with teaching, and learning efficiency, with a statistically significant difference. On the other hand, the differences in improving self-directed learning ability, communication ability, and hands-on ability were not statistically significant.

Table 3 Effects of interventions assessed by questionnaires

Sensitivity analysis

Owing to high heterogeneity, sensitivity analysis was implemented to evaluate the reliability of the results. After excluding the study with the largest weight [24], the pooled effect size was in favor of the intervention group (WMD 7.03, 95% CI: 5.58–8.50, P < 0.00001) for KS and did not change the effects observed in the primary analysis. Conversely, after excluding the study with the largest weight [24, 27, 81], the pooled effect size in SS, PS assessment, and TS was in favor of the intervention group (WMD 6.28, 95% CI: 4.33–8.22, P < 0.00001; WMD 8.45, 95% CI: 7.08–9.82, P < 0.00001; WMD 8.61, 95% CI: 6.56–10.67, P < 0.00001). No single study was found to significantly influence the overall pooled WMD, indicating the stability of our results.

Publication bias

The evaluation of publication bias was conducted using a funnel plot for the 27 studies with respect to the total scores (Fig. 7). The shape of the funnel plot did not show asymmetry, indicating the absence of any publication bias. Also, no significant bias was detected using the Begg’s rank correlation test (Z = 0.21, P = 0.835) and Egger’s linear regression test (t = -1.23, P = 0.228).

Fig. 7
figure 7

Funnel plot analysis for total scores

Discussion

In the current meta-analysis, the results showed that the residents in the PBL-based teaching groups have better scores in knowledge, skill, PS assessments, and TS than those in LBL groups, indicating that PBL could help residents to better master the medical theory knowledge, operational skills, analyze and diagnose cases and overall capacity than LBL. The QS showed that PBL-based strategies are superior to LBL in improving residents’ theoretical knowledge mastery, clinical diagnostic thinking, teamwork ability, ability to analyze and solve problems, ability to consult documents, learning interest, and learning efficiency. Also, the residents exhibited more satisfaction with teaching for PBL-based strategies than for LBL. However, PBL-based strategies did not significantly improve self-directed learning ability, communication ability, and hands-on ability.

Clinical cases are used as problems of PBL in medical education

PBL and CBL are student-centered, focusing on students’ subjective initiative, cultivating their ability to find and solve problems. However, they are different teaching strategies since PBL commonly presents complex, open-ended problems about topics previously unknown to the students and develops problem-solving skills through self-teaching and discussion, even solutions may vary from group to group. However, CBL takes clinical cases as the starting point, uses relevant knowledge and theory to analyze these cases, solves the clinical problems efficiently, and improves the clinical ability. Over the years, PBL in medical education has shifted towards a case-focused approach, wherein the only problems considered by the students are patient cases, and the discussion phase is reduced to a minimalistic list of questions or learning objectives, and reporting diagnoses and medical facts is emphasized. This method prompted the medical educators to wonder about the purpose of PBL and switch to the CBL method. Other educators using the term CBL referred to a case as the problem [94, 95]. Thus, all the studies in this meta-analysis encompassed clinical cases as trigger problems. To avoid incomplete inclusion of the literature due to differences in understanding of the concepts and to accurately assess the role of PBL teaching methods in SRT, we included studies involving PBL or PBL combined with CBL. Also, heterogeneity analysis was carried out according to the teaching method, but the results did not reduce heterogeneity (see Sect. 4.3 for details).

Analysis of the role of PBL in the cultivation of residents’ abilities

Thomas et al. [96] found that residents who attended a PBL medical school (PBL group) performed significantly better on standardized tests than on those who attended a traditional medical school in obstetrics and gynecology residency program, which is consistent with our finding of knowledge acquisition. Sun et al. [97] found that compared to the traditional teaching model, problem- and simulator-based learning for lumbar puncture training can develop overall surgical skills in neurology residents, which is consistent with the current finding about operational skills. In subgroup analysis, we found that PBL based teaching shows no advantage on operational skills in diagnostic imaging. In another study, Yue et al. [98] found that the integration of PBL and LBL teaching modes in the education of imaging diagnosis education produced a good teaching effect, which needs to be substantiated with additional studies. The residents’ ability to analyze and diagnose cases include history taking, physical examination, and analysis of inspection results is crucial for residency training. According to our findings in PS assessments, PBL-based teaching enabled the development of this critical capability. The above analysis of different capabilities showed overall positive capacity according to TS.

The residents showed a preference for PBL-based strategies. The positive effect of PBL on mastery of theoretical knowledge, clinical diagnostic thinking, teamwork ability obtained by our analysis which is in agreement with previous studies may explain this result well [99, 100]. The survey of self-directed learning showed some improvement in residents, but the improvement is not significant in the PBL group. The reason for this could possibly be associated with residents dealing with various problems of patients every day, has developed stronger self-directed learing ability in clinical practice than undergraduates. Zhang et al. [61] discussed that the PBL group requires time and effort for preparation before the class, while the existing teaching facilities could not fully meet their needs. In addition, the students in China received “spoon-feeding” education for a long time, and the literature retrieval level was limited. Therefore, improving the level of teaching hardware, and giving guidance on the retrieval methods is needed. The PBL teaching mode could not improve communication ability, and the analysis by Sanghee et al. [101] might explain the related factors, because of the cultural climate of Asian countries, students were reluctant to express their opinion to a tutor who has authority and felt uncomfortable to challenge classmates’ views. It’s worth noting that only three included studies assessed the communication ability as the intervention outcome measures. This may imply that insufficient attention was paid to this ability when conducting PBL in Chinese SRT. Therefore, the guidance and encouragement of the tutor is necessary for the development of effective communication, not only between the resident and the tutor, but also among the training residents.

Analysis of heterogeneity

An obvious heterogeneity was detected among the included studies for KS, SS, PS assessments, and TS. In order to explore the source of heterogeneity, the teaching methods and departments were analyzed in subgroups, but the results did not show reduced heterogeneity. The reasons for the analysis of high heterogeneity are as follows.

First, the comprehensive ability of the resident teacher is an important contributor to the training of qualified residents, as well as a factor related to the quality of SRT. The difference in the teaching level exerts an influence on the teaching effect. Although all the studies in this meta-analysis are carried out in hospitals affiliated to medical schools or equivalent providing high-level medical and health services, most studies do not mention the situation of the teachers. In addition, teachers should strengthen the study of new theories and methods. Although teachers are familiar with PBL teaching mode with an extensive attempt of PBL in undergraduate teaching, only a few described that teachers had received the PBL teaching training. The existing studies also lack the supervision and evaluation of teachers. The difference in the teaching level of teachers may be one of the causes of heterogeneity.

Second, the residents' basic quality and learning experience are different [102]. Some are SRT trainees applying for Master of Medicine degree during the 3 years simultaneously, which might have a strong learning aspiration and ability. Only a few studies mentioned previous clinical working hours and whether residents had been exposed to PBL teaching methods. Therefore, we deduced that the learning ability, clinical work experience, and PBL training experience of residents are the potential causes of heterogeneity.

Third, the present evaluation method formulated by the training department could not form a unified evaluation system for each specialty according to its characteristics. In addition, for capacity assessment, a long-term evaluation should be more suitable. Carrero et al. [103] found that the effectiveness of lecture and case/problem-based learning differed only slightly in terms of improving immediate clinical capacity in the first year Anesthesiology residents while suggesting that there should be an appropriate tool to determine the effect of different teaching methodologies on the long-term retention of knowledge, skills, attitudes and clinical competence. Therefore, lack of a unified evaluation system may be one cause for heterogeneity.

Limitations

In addition to the factors mentioned above that may cause heterogeneity, the quality of the articles is also one of the limitations of this meta-analysis. Although all the studies included in this meta-analysis were randomized controlled trials, none of them described the allocation concealment in detail, and no blinding method was used which was caused by the objective limitations in teaching. The small sample size in some of the studies was also one of the limitations. Besides, at present, there is no unified scale to assess the above mentioned skills of residents in the included studies. For example, QS in each study were designed by the training department itself based on the contents of previous reported questionnaires and the goal of SRT. So the inconsistencies of the assessment among these included studies were also one of the limitations. Another limitation about QS is that because the results are assessed in a yes or no form, resulting in missing data and inefficient data use. Additionally, there were many possible factors influencing the effect of PBL, stratified analysis should be conducted to test the heterogeneity.

Conclusions

The present meta-analysis shows that the PBL teaching method is more effective than LBL in the mastery of medical theory knowledge, operational skills, analysis and diagnosis of cases, and the overall capacity of SRT in mainland China. However, it shows no advantage on operational skills in diagnostic imaging. QS displayed the positive effects of PBL on theoretical knowledge mastery, clinical diagnostic thinking, teamwork ability, ability to analyze and solve problems, ability to consult documents, learning interest, and learning efficiency. The QS results also indicated that residents showed more satisfaction with teaching for PBL-based strategies than for LBL. However, PBL-based strategies had not improved significantly with respect to self-directed learning ability, communication ability, and hands-on ability. This meta-analysis provided a systematic and comprehensive analysis and achieved the training contents suitable for the PBL teaching model. To the aspects which the results did not show any improvement, the effect of other teaching methods should be discussed in the future. According to our results, the basic data could be obtained for a detailed design and performance of suitable teaching methods for various specialties and abilities in Chinese SRT. Nonetheless, the heterogeneity of the included studies needs to be considered, and further well-designed studies are needed to confirm our findings.

Availability of data and materials

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

Abbreviations

SRT:

Standardized residency training

PBL:

Problem-based learning

LBL:

Lecture-based learning

CBL:

Case-based learning

QS:

Questionnaire surveys

KS:

Knowledge scores

SS:

Skill scores

PS:

Practical skills

TS:

Total scores

EB:

Educational background

DS:

Department entrance exam scores

PT:

Professional titles

TE:

Teaching experience

CLP:

Collective lesson preparation

References

  1. Zhu JM, Li WK, Chen L. Doctors in China: improving quality through modernisation of residency education. Lancet. 2016;388(10054):1922–9.

    Article  Google Scholar 

  2. Zhang Q, Lee LM, Gruppen LD, Ba DN. Medical education: changes and perspectives. Med Teach. 2013;35(8):621–7.

    Article  Google Scholar 

  3. Hou JL, Michaud C, Li ZH, Dong Z, Sun BZ, Zhang JH, Cao D, Wan XH, Zeng C, Wei B, Tao LJ, Li XS, Wang WM, Lu YQ, Xia XL, Guo GF, Zhang ZY, Cao YF, Guan YZ, Meng QY, Wang Q, Zhao YH, Liu HP, Lin HQ, Ke Y, Chen L. Transformation of the education of health professionals in China: progress and challenges. Lancet. 2014;384(9945):819–27.

    Article  Google Scholar 

  4. Cao ZZ, Wang L. China’s Evolving Residency Training. Med Teach. 2015;37(9):884–5.

    Article  Google Scholar 

  5. Wen DL, Ma XD, Li HH, Xian BS. Empathy in Chinese Physicians: Preliminary Psychometrics of the Jefferson Scale of Physician Empathy (JSPE). Med Teach. 2013;35(7):609–10.

    Article  Google Scholar 

  6. Deng GW, Qian J, Zhang XQ, Xu HQ. The neglect in the diagnostic process of Chinese doctors-communication and interpersonal skills. Med Teach. 2014;36(12):1089.

    Article  Google Scholar 

  7. Hung W. The 9-step problem design process for problem-based learning: application of the 3C3R model. Educ Res Rev. 2009;4(2):118–41.

    Article  Google Scholar 

  8. Dring JC. Problem-based learning - experiencing and understanding the prominence during medical school: perspective. Ann Med Surg (Lond). 2019;47:27–8.

    Article  Google Scholar 

  9. McCarthy MK, Birnbaum RJ, Bures J. Problem-based Learning and Psychiatry Residency Education. Harv Rev Psychiatry. 2000;7(5):305–8.

    Article  Google Scholar 

  10. Ma J, Guo JH, Zhang T, Ye W, Zhou Y, Qi CW, Wu JJ. Meta-analysis of learning outcomes with problem-based learing teaching method in clinical talent standardization training. Transl Med J. 2017;6(5):280–90.

    Google Scholar 

  11. Liu J, Zhao G, Meng T, Song G. System evaluation of the application of PBL and CBL in resident training. Health Vocational Educ. 2018;36(24):144–7 (In Chinese).

    Google Scholar 

  12. Lio J, Ye YQ, Dong HM, Reddy S, McConville J, Sherer R. Standardized Residency Training in China: The New Internal Medicine Curriculum. Perspect Med Educ. 2018;7(1):50–3.

    Article  Google Scholar 

  13. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JAC, Cochrane Bias Methods Group, Cochrane Statistical Methods Group. The Cochrane Collaboration’s Tool for Assessing Risk of Bias in Randomised Trials. BMJ. 2011;343:d5928.

    Article  Google Scholar 

  14. Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring Inconsistency in Meta-Analyses. BMJ. 2003;327(7414):557–60.

    Article  Google Scholar 

  15. DerSimonian R, Laird N. Meta-Analysis in Clinical Trials. Control Clin Trials. 1986;7(3):177–88.

    Article  Google Scholar 

  16. Mantel N, Haenszel W. Statistical Aspects of the Analysis of Data from Retrospective Studies of Disease. J Natl Cancer Inst. 1959;22(4):719–48.

    Google Scholar 

  17. Begg CB, Mazumdar M. Operating Characteristics of a Rank Correlation Test for Publication Bias. Biometrics. 1994;50(4):1088–101.

    Article  Google Scholar 

  18. Egger M, Smith GD, Schneider M, Minder C. Bias in Meta-Analysis Detected by a Simple. Graphical Test BMJ. 1997;315(7109):629–34.

    Google Scholar 

  19. Li M, Liang B, Li YW, Yang QY, Gao F. Application of PBL teaching method in the teaching rounds of general practice residency training. China Higher Medical Education. 2019;4:100-101. (In Chinese).

  20. Wang N, Yan W, Yan T. Application of PBL teaching method in internal medicine base for standardized residency training. Research analysis. 2017;13:324. (In Chinese).

  21. Gao WQ, Wang LB, Yang GY, Zhao XH, Li YH. Application of PBL teaching mode in cardiovascular skill training of general residents. Medical Information. 2019;32(7):20-21. (In Chinese).

  22. Liu J, Ling Y, Wang J, Xi CH, Yuan C. Application of PBL teaching method in resident standardized training in the Department of Gastroenterology. China Modern Doctor. 2017;55(33):141-144. (In Chinese). 

  23. Gulina A, Wang XM. Discussion on the application value of PBL teaching method in standardized residency training in gastroenterology. Healthful Friend. 2019;10:154. (In Chinese).

  24. Wei DM, He GN, Yang WJ. Effect of applying PBL teaching method in standardized residency training in gastroenterology. Chin J Rural Med Pharm. 2017;24(3):67–8 (In Chinese).

    Google Scholar 

  25. Hou H, Liu HM, Liu J. Analysis of the application effect of applying PBL teaching in the standardized residency training in gastroenterology. China Health Care & Nutrition. 2019; 13:394. (In Chinese).

  26. Chang BC, Chen WD, Zhang Y, Wu XP, Guo YL. Evaluate the teaching effect of PBL by application Mini-CEX in resident standardization training of chief physician. Journal of Qiqihar Medical University. 2018;39(5):583-585. (In Chinese).

  27. Tang JL, Shi M, Huang CL, Li XM, Zhang L. Application of WeChat and PBL teaching in general practice training. Continuing Med Educ. 2019;33(10):17–9 (In Chinese).

    Google Scholar 

  28. Wang YY, Zhou QY, Zhu LY, Zhong LH, Lu BL, Cheng Y, Yao H, Fan J, Yu L. The influence of teaching model of WeChat combined with the PBL to the teaching quality of standardized training of residents. China Continuing Medical Education. 2017;9(15):10-12. (In Chinese).

  29. Xie SS, Zhang Y, Zuo WZ. The application of PBL teaching method in the standardized residency training in the Infectious Disease Department. J Mod Med Health. 2018;34(7):1103-1105. (In Chinese).

  30. Zhao D. Research on the application of PBL teaching model based on diagnosis and treatment guidelines to the standardized residency training in Intensive Care Medicine. Health Educ. 2019;21:123-124. (In Chinese).

  31. Xing JY, Han XN, Yuan ZY, Sun XX, Xu XF, Sun YB. Application of clinical cases related problem-based-learning in critical care medical education. Chin J Med Edu Res. 2017;16(6):614-618. (In Chinese).

  32. Liu ZG, Gan J, Wang XJ. Practice of PBL teaching method in rotation teaching of the Department of Neurology. China Higher Medical Education. 2012;11:20-21,77. (In Chinese).

  33. Wang Y. Application of PBL combined with LBLm teaching model in standardized residency training of neurology. Anhui Medical Journal. 2015;36(7):891-893. (In Chinese).

  34. Huang YX. Application of PBL teaching method in standardized training of neurological residents. China Continuing Medical Education. 2019;11(31):67-69. (In Chinese).

  35. Shi JQ, Jiang T. The application of problem-based learning education model in the standardized training for neurology residents and its effect evaluation. Journal of Jiangsu Institute of Commerce. 2018;6:59-61. (In Chinese).

  36. Cheng J, Chen W, Gan J, Liu ZG. Practice of PBL teaching method in the teaching of resident neurology in different educational backgrounds. China Higher Medical Education. 2015;2:94-95. (In Chinese).

  37. Huang JX, Zhu HM, Wu LQ, Zhang YN. Application of PBL teaching mode in standardized residency training of emergency. Journal of Traditional Chinese Medicine Management. 2016;24(23):84-85. (In Chinese).

  38. Lin F, Yu QY, Zheng JH, Pan XX, Hang JG. Application of PBL teaching mode on standardized training of pediatric residents. Hospital Management Forum. 2016;33(10):53-55. (In Chinese). 

  39. Jiang CQ, Zhu PS, Shi Y, Xiang WJ, Ge ST, Zhang ZB, Zuo LG. Effect analysis of using PBL teaching mode in resident standardization training for professional postgraduate students majored in general surgery. Journal of Qiqihar Medical University. 2018;39(21):2554-2557. (In Chinese).

  40. Huang XX. The application of PBL teaching model in standardized training for resident in gastrointestinal surgery. Journal of Anhui Health Vocational & Technical College. 2018;17(6):109-110,113. (In Chinese).

  41. Ge ST, Shao RR, Zhang ZB, Liu ML, Jiang CQ. Application of problem-based learning educational model in resident standardization training of gastrointestinal surgery. China Continuing Medical Education. 2018;10(7):1-3. (In Chinese).

  42. Guan YB, Nong SJ, Cai B, Ma LM, Xing QW, Geng JS. A practical study on PBL teaching method in standardized training of urology residents. Continuing Medical Education. 2018;32(12):38-40. (In Chinese).

  43. Zhang JL, Lu QZ, Zhang ZY, Hou JQ. The analysis of efficacy of PBL teaching method in standardization training of residents in urology. Education Teaching Forum. 2019;50:209-210. (In Chinese).

  44. Ma Y, Liu ZW, Zhu X, Ma JY, Sui JT, Li Y. Application of PDCA cycle combined with PBL teaching method in training of orthopedics residents. Journal of Bingtuan Medicine. 2018;3:64-66. (In Chinese).

  45. Zhou P, Wang XQ, Li XY, Li ST. Application of PBL teaching model in standardized training of neurosurgery clinical base. Education Teaching Forum. 2014;33:107-108. (In Chinese). 

  46. Lin Y, Jiang H. Application of problem-based learning as a teaching model in difficult airway management training using a simulator. Chinese Journal of Medicinal Guide. 2014;16(6):1103-1104. (In Chinese).

  47. Li ZR, Geng XB, Sun ML, Ma HN, Liu M, Zhou CY. Comparison of the effects of PBL and LBL in standardized residency training of anesthesiology. Our Health. 2019;12:123. (In Chinese).

  48. Jiang J, Lin Y, Zheng YC, Huang Y, Jiang H. Problem-based learning versus lecture-based learning in standardized training of residents in anesthesiology. Chinese Medical Record. 2017;18(10):98-101. (In Chinese).

  49. Xin WQ, Yan Z, Shi CN, Guo SL, Li AX, Zhang YX. Effect of WeChat platform combined with problem-based learning teaching method when applied to the standardized training for residents in a Department of Anesthesiology. Chin J Anesthesiol. 2017;37(4):392-395. (In Chinese).

  50. Zheng LJ, Guo LS. Application of PBL teaching methods in the standardized resident training of obstetrics and gynecology. China Continuing Medical Education. 2018;10(18):5-7. (In Chinese).

  51. Han J, Yan XL. Comparisons of teaching methods in standardized training for ophthalmic residents. Int Eye Sci. 2017;17(5):904-907. (In Chinese).

  52. Chen JL. The application value of three teaching modes of PBL, TBL and LBL in the standardized training of ophthalmology residents. Contemporary Medical Symposium. 2018;16(19):192-194. (In Chinese). 

  53. Liu GX, Zhong AX, Pan XB, Peng LH, Lu QF, Pan MZ, Li RT. The application and thinking of PBL teaching model in the standardization training of resident in clinical prosthodontics. Journal of Minimally Invasive Medicine. 2018;13(5):665-667. (In Chinese).

  54. Chen HB, Ma Q, Liu ST. Application of PBL teaching method in standardized training of ultrasonic medical residents. China Continuing Medical Education. 2019;11(2):1-2. (In Chinese).

  55. Yang JC, Ma J, Wang P, Wang L, Lu JS, Du L. Application of PBL teaching method in the clinical teaching of resident doctors in Ultrasound Medicine Department. Chinese Medical Record. 2015;16(9):79-81. (In Chinese).

  56. Fan X, Song T, Hou XQ, Dong J, Yang LF. Application of PBL teaching mode in standardized residency training in ultrasonic base. Health Vocational Education. 2016;34(10):73-74. (In Chinese).

  57. Dong FL, Fan QM. The application of PBL teaching mode in the standardization training of residents in ultrasound medicine. Fork Lore. 2015;123-124. (In Chinese).

  58. Lu Y, Yu DH, Zhang B, Zhang HZ, Wang MH. Application of problem-based learning model in general practical standard training. GP Chinese General Practice. 2014;17(16):1880-1883. (In Chinese).

  59. Wang Y, Deng BC, Liu P. The application of PBL in resident standardization training in the Department of Infectious Diseases. China Continuing Medical Education. 2018;10(14):9-11. (In Chinese).

  60. Yi XL, Li XL, Jiang Y. Application of PBL teaching mode in standardized training of pediatric residents. Journal of Qiqihar University of Medicine. 2017;38(16):1936-1937. (In Chinese). 

  61. Zhang J, Yang KP, Hou XS, Ge XQ, Qiao MH, Hou L. The applying of problem-based learning in standardized training of residents of thoracic surgery. Chin J Med Edu. 2017;37(5):779-782,796.

    Google Scholar 

  62. Chen JX, An HY, Wang F. Research of PBL teaching method in urology surgical practice. Chinese Medicine Modern Distance Education of China. 2015;13(4):111-112. (In Chinese).

  63. Ma Y, Zhang X. Application effect of PBL teaching in standardized training of ophthalmology residents. Jilin Medical Journal. 2019;40(3):679-680. (In Chinese).

  64. Wang Z, Su DK, Lai SL, Jin GQ, Xie D, Kang W, Zhao Y, Yang J. The application of PBL teaching method in the standardized training of residents in oncology medical imaging. Guangxi Medical Journal. 2019;41(12):1597-1599. (In Chinese).

  65. Yang XY, Jia F. The application of CBL combined with PBL method in the ward round teaching of residents in Department of Cardiology. China Continuing Medical Education. 2019;11(36):24-26. (In Chinese).

  66. Jiang H, Najina W, Wang K, Wang LP, Ailiman M. Application of PBL and CBL dual-track teaching in standardized training of residents in cardiology. Xinjiang Medical Journal. 2017;47(6):686-687. (In Chinese).

  67. Lin FN, Huang LH, Xiong SQ, Zheng F, Zhan P, Li CY, Lin W, Qiu CY. Application of PBL and CBL combined teaching method in the diagnosis and treatment of coronary heart disease. Chinese Journal of Geriatric Care. 2017;15(3):126-127. (In Chinese).

  68. Shi XJ, Liang XP, Yi He lasi, He FP, Fan XT. Evaluation of the effect of CBL combined with PBL teaching model in standardized training of resident doctors. Medical Information. 2018;31(6);16-18. (In Chinese). 

  69. Jin L, Fang L, Li SY, Zhu WC, Dai FL, Dun SS. The Application of PBL and CBL combined with holistic integrated medicine in the training of digestive endoscopy. China Continuing Medical Education. 2018;10(30):3-6. (In Chinese).

  70. Hu XL, Chen WD, Sun WH, Yu L, Shi ZM, Jin GX. Application of the teaching method combining PBL with CBL in resident standardization training in the Department of Endocrinology. Chinese Journal of General Practice. 2017;15(7):1236-1238. (In Chinese).

  71. Wu Y, Zhang M, Wang XC, Zhao YJ. The application of problem/case-based learning teaching method in standardized training of residents in the Department of Critical Care Medicine. China Continuing Medical Education. 2018;10(29):16-19. (In Chinese).

  72. Li SX, Yin LL, Wang M, Wang LL, Zhao S, Zhao JL. The use of PBL combined with CBL in emergency resident standardized training. China Continuing Medical Education. 2019;11(34):28-30. (In Chinese).

  73. Liu FS, Wang SM, Liu J, Zheng XC, Li SF, Zhu Y, Shen W, Pan HH, Fang XL. The application effect of CBL combined with PBL teaching method oriented by post competence in clinical teaching of traditional Chinese medicine emergency. China Higher Medical Education. 2018;6:81-82. (In Chinese).

  74. Song YH. Application of two teaching methods in the training of psychiatric residents. China Continuing Medical Education. 2019;11(33):48-50. (In Chinese).

  75. Guan XL, Zheng Y, Bi S, Lu XL, Yang H, Tao Y, Hu J. Application of PBL combined with CBL teaching method in the multiple sclerosis teaching for training of residents. Studies of Trace Elements and Health. 2018;35(3):71-73. (In Chinese). 

  76. Liu HH, Xiao GD. Application of CBL combined with PBL teaching model in clinical standardized training of neurology residents. Chinese Journal of Coal Industry Medicine. 2018;21(5):553-556. (In Chinese).

  77. Wan QQ, Yan PL, Wu F. Discussion on teaching methods of tracheal intubation in pediatric resident standardized training. China Modern Doctor. 2018;56(28):132-134. (In Chinese).

  78. Zhao Y, Zhang CX, Guo YY. Application of PBL combined with medical record in the standardized training of pediatric resident physicians. Medicine and Health Education. 2017;23:60-61. (In Chinese).

  79. Li H, Li SL, Li YM, Liu QF, Liu Y. Analysis of the effect of CBL, PBL combined with simulation training method in clinical teaching of general surgery. China Continuing Medical Education. 2019;11(18):16-18. (In Chinese).

  80. Shi GY. Application of PBL combined with CBL teaching method in standardized training of general surgery residents. Health Must-Read Magazine. 2019;35:261. (In Chinese).

  81. Liu F, He JG. Application combined PBL with CBL in the standardized training of resident physicians in the Digestive Diseases Department. J Nongken Med. 2018;40:76–9 (In Chinese).

    Google Scholar 

  82. Hu GD. Application of CBL and PBL combined with simulated teaching method in the standardized training of residents in Cardiothoracic Surgery Department. China Health Care & Nutrition. 2019;29(27):316-317. (In Chinese).

  83. Zhang L, Tang Z, Liang B, Li XJ, Gong HY, Wang B. Application of CBL and PBL combined with simulated training method in the teaching of clinical resident physician in Cardiothoracic Surgery department. Chinese Journal of General Practice. 2017;15(7):1233-1235. (In Chinese).

  84. Xu L, Xiao SW, You CY, Tang HB, Zhang XJ, Wu HT, Zhao HX, Dai Y, Chen XZ, Xie MX. Research and analysis on the teaching methods of standardized training for surgical residents in neurosurgery rotation. Medicine and Health Education. 2017;14:92-93,96. (In Chinese).

  85. Zhao L, Du R, Ma R, Ma CL. The effectiveness of applying PBL combined with CBL teaching in the standardized training of residents in obstetrics and gynecology. International Infectious Disease (Electronic Edition). 2019;8(3):224-225. (In Chinese).

  86. Ji H. The application of PBL and CBL double track teaching in standardized training of obstetrics and gynecology resident. China Continuing Medical Education. 2018;10(7):3-5. (In Chinese).

  87. Jiang HJ, Feng GL, Jia YL, Jiang H, Zhang MY, Xu HL. Application of PBL combined with CBL teaching model in standardized training of residents in Radiology Department. China Higher Medical Education. 2019;2:108-109. (In Chinese).

  88. Wang HY, Kong LL. Application of CBL and PBL teaching methods in the teaching of residents receiving standardized training in the Tumor Radiotherapy Department. The Science Education Arrical Collects. 2019;465:100-102. (In Chinese).

  89. Xiao L, Mayinuer A, Bao YX. Application of problem-based learning combined with case based learning in standardized training of residents in Department of Radiation Oncology. Chinese Medical Record. 2018;19(8):76-78. (In Chinese). 

  90. Chen Z, Xu L, Jin Y, Kou ZJ, Wang XD. A study on the application of PBL combined with CBL in standardized training for residents of Tuina Department. Journal of New Chinese Medicine. 2019;51(7):318-320. (In Chinese).

  91. Wang Y, Hao W. Effect evaluation of CBL combined with PBL teaching mode in the clinical teaching of Cardiovascular Department. Chinese Medical Record. 2017;18(9):83-85. (In Chinese).

  92. Wang BQ, Xue F, Zhang GQ, Ding W. Application of teaching method based on questions and cases in the training of residents in oncology surgery. China Higher Education. 2016;5:67-68. (In Chinese).

  93. Xu P, Li CJ. Application of PBL and CBL teaching in the standardized training of oral resident. China Continuing Medical Education. 2016;8(33):3-4. (In Chinese).

  94. Servant-Miklos VFC. The Harvard Connection: How the Case Method Spawned Problem-Based Learning at McMaster University. Health Prof Educ. 2019;5(3):163–71.

    Article  Google Scholar 

  95. Kulak V, Newton G. A Guide to Using Case-Based Learning in Biochemistry Education. Biochem Mol Biol Educ. 2014;42(6):457–73.

    Article  Google Scholar 

  96. Thomas J, Aeby T, Kamikawa G, Kaneshiro B. Problem based learning and academic performance in residency. Hawaii Med J. 2009;68(10):246–8.

    Google Scholar 

  97. Sun CJ, Qi XK. Evaluation of Problem- and Simulator-Based Learning in Lumbar Puncture in Adult Neurology Residency Training. World Neurosurg. 2018;109:e807–11.

    Article  Google Scholar 

  98. Yue JY, Chen J, Dou WG, Liang CH, Wu QW, Ma YY, Zhu ZP, Li MX, Hu YL. Using integrated problem- and lecture-based learning teaching modes for imaging diagnosis education. BMC Med Educ. 2018;18(1):183.

    Article  Google Scholar 

  99. Ibrahim ME, Al-Shahrani AM, Abdalla ME, Abubaker IM, Mohamed ME. The Effectiveness of Problem-based Learning in Acquisition of Knowledge, Soft Skills During Basic and Preclinical Sciences: Medical Students’ Points of View. Acta Inform Med. 2018;26(2):119–24.

    Article  Google Scholar 

  100. Dan P, Ni J, Song D, Zhang W, Wang Y, Liling W, Wang X, Wang Y. Influence of critical thinking disposition on the learning efficiency of problem-based learning in undergraduate medical students. BMC Med Educ. 2019;19:1.

  101. Yeo S, Chang BH. Implementation of problem-based learning in medical education in Korea. Korean J Med Educ. 2017;29(4):271–82.

    Article  Google Scholar 

  102. Xiao YT, Wang YT. Standardised Residency Training: Students’ Concerns. Lancet. 2017;389(10072):905.

    Article  Google Scholar 

  103. Carrero E, Gomar C, Penzo W, Rull M. Comparison Between Lecture-Based Approach and case/problem-based Learning Discussion for Teaching Pre-Anaesthetic Assessment. Eur J Anaesthesiol. 2007;24(12):1008–15.

    Article  Google Scholar 

Download references

Acknowledgements

We thank Ms. Cong Ai for her guidance on the data analysis.

Funding

This study was funded by the National Natural Science Foundation of China (NSFC) [Grant No.81700637], the Teaching Reform Foundation of Liaoning Province [Grant No. 20042088] and the Natural Science Foundation of Liaoning Province [Grant No. 20170540993].

Author information

Authors and Affiliations

Authors

Contributions

XL designed the study, obtained funding and oversaw the project. WW, ZL and HW collected the data. TL and XL analyzed data and wrote the manuscript. The authors reviewed and approved the manuscript.

Corresponding author

Correspondence to Xiaodan Liu.

Ethics declarations

Ethics approval and consent to participate

Not applicable (This manuscript is a meta-analysis, and does not report on or involve the use of any animal or human data or tissue.)

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Table A1.

The references of included studies.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, T., Wang, W., Li, Z. et al. Problem-based or lecture-based learning, old topic in the new field: a meta-analysis on the effects of PBL teaching method in Chinese standardized residency training. BMC Med Educ 22, 221 (2022). https://doi.org/10.1186/s12909-022-03254-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12909-022-03254-5

Keywords