Developing the Comprehensive Medical Professionalism Assessment Scale

Background: There has been an increasing number of articles that have studied the topic of medical professionalism. The aims of this study were to develop a scale of medical professionalism. Methods: The concepts of medical professionalism as defined by associations and groups of physicians, nurses and physiotherapists were investigated. The surveys using self-administered questionnaires were conducted on students and junior residents. An exploratory factor analysis, calculation of the coefficient alpha, and correlation coefficients with other scales were performed. Results: A factor analysis resulted in the extraction of 30 items in 7 factors as items of the Level 1 scale for preclinical level students. The correlation coefficients between the scores for the 7 factors and KiSS-18 were in the range of 0.23 0.76. The coefficient alpha of all 30 items was 0.90. A factor analysis resulted in the extraction of 31 items in 8 factors from the Level 2 scale for students at the time of graduation. The correlation coefficient between the scores for the 8 factor and "Reflective skills" of the P-MEX was 0.31 0.59. The coefficient alpha for all 30 items was 0.93. Conclusion: Construct validity, criterion-related validity and reliability were generally confirmed for the two scales.


Introduction
There have been an increasing number of studies on medical professionalism since the 1990s (Smith, 2005). Traditionally, professionals such as physicians and attorneys were considered professionals based on the social processes through which the professions were established. However, it has been criticised that physicians are not sufficiently fulfilling the social contracts (Hafferty, 2009), and physicians are consequently expected to cultivate higher levels of professionalism.
There are some definitions of professionalism. Stern and Arnold (Stern, 2006) stated that "professionalism is demonstrated through a foundation of clinical competence, communication skills, and ethical and legal understanding, upon which is built the aspiration to and wise application of the principles of professionalism". The Medical Professionalism Project also defined three fundamental principles and ten professional responsibilities of professionalism as the Physician Charter (ABIMFoundation et al., 2002). A comparison between the concepts of professionalism by Arnold and Stern(Stern, 2006) and the Physician Charter (ABIMFoundation et al., 2002) results in many common and divergent aspects. Therefore, the development and assessment of programmes that foster medical professionalism first requires defining medical professionalism and then measuring it.
The most commonly used instrument to measure professionalism is that created by Arnold et al. (Arnold et al., 1998) based on the definitions of professionalism given by the American Board of Internal Medicine (ABIM) in 1995, and several studies that revised and used this instrument. However, the results of factor analysis showed that there were some items with higher factor loading among several factors, and a reliability assessment showed that some subscale items had a coefficient alpha of less than 0.6. As Arnold et al. (Arnold et al., 1998) suggested the necessity of adding questionnaire items, this scale lacks reliability and validity.
The concept of professionalism has changed over time, from how the ABIM defined the concept of professionalism in 1995 to that published in the Physician Charter in 2002. The original six elements of professionalism presented in 1995 are no longer adequate for describing the concept of professionalism. Many other scales and instruments for measuring and evaluating professionalism of physicians have been developed, but there are many that have not been tested for construct validity.
This study seeks to develop a medical professionalism assessment scale for evaluating how medical professionalism is taught and fostered in students. Of the presumed steps to acquire medical professionalism, this study defines the stage before starting clinical practice as Level 1 and the stage at the time of graduation as Level 2, and develops two scales matched to the students' level of readiness. Development of these scales will contribute to objective evaluation of the outcomes of education in medical professionalism.

Methods
The following procedures were undertaken for developing an assessment scale of medical professionalism. First, the concepts of medical professionalism is extracted. Second, an item pool will be created based on the concepts extracted as a source for the scale items. Third, a pre-test will be conducted to develop the Level 1 scale for assessment readiness in pre-clinical students. Fourth, the main survey for developing the Level 1 scale for assessment of readiness in pre-clinical students will be conducted. Fifth, the Level 2 scale adapted for assessing the readiness of students at the time of graduation will be developed.
Step 1: Extracting the concepts of medical professionalism A close investigation was performed on various documents (Table 1)  physicians, nurses and physiotherapists as presented or published by associations of each profession in the four countries of Japan, the United States, United Kingdom and Canada. Content corresponding to medical professionalism were coded and abstraction and specification of concepts were repeated until 90 codes in 21 categories, 7 areas and 3 fields on the concept of medical professionalism were extracted (Table 2). Interactions between the patient, the medical profession and other health professionals (2) Patient-centred care (21) Patient-centred approach (21a) Supporting patient autonomy (21b) Understanding and interacting with the patient as a whole person (21c) Collaborative practice (22) Collaboration with professionals in other fields (22a) Collaboration within the medical/healthcare profession (22b) Building an organizational environment for care (23) Preparing the organizational environment for care (23a) Providing and promoting safe medical care (23b) Fulfil social responsibilities (3) Contributing to the community, professional associations and society (31) Affiliation and contributions to professional medical/healthcare associations (31a) Understanding of and contribution to public health activities in the community (31b) Contributing to health policies (31c) Use of mass media and information provision (31d) Understand the general legal and ethical principles to fulfil social responsibilities(32) Respect legal and ethical principles (32a) Appropriate handling and protection of personal information (32b) Fulfilling social responsibilities (32c) Concept identification numbers in parentheses. Ninety "code" subordinated to the "category" are omitted due to the lack of space.

Step 2: Investigating the concepts of medical professionalism and creating an item pool
Based on the concepts of medical professionalism, a working group of seven individuals, including the author made a close investigation of the draft items and created an item pool composed of 259 items.
Step 3: Developing the Level 1 Scale: Pre-test Study design: A cross-sectional study with a collective survey using anonymous, self-administered questionnaires were conducted in April -June 2016.
Subjects: Subjects were first-year students of two universities located in Hokkaido, the northernmost part of Japan. The survey was conducted on 714 first-year students in eight departments (pharmacy, dentistry, nursing, clinical social welfare, clinical psychology, physical therapy, occupational therapy, speech-language-hearing therapy and dental hygiene) of University A, and 110 first-year students of the department of medicine of University B.
Survey items: As demographic characteristics, students' academic faculty and department, year, and gender were surveyed. Thirty-five items were selected from the item pool as items to assess medical professionalism. The response options were as follows: Agree' (5 points), 'Agree to a certain extent' (4 points), 'Neither' (3 points), 'Disagree to a certain extent' (2 points), and 'Disagree' (1 point).
Analysis: Exploratory factor analysis was performed to confirm the factor structure of the concepts of medical professionalism. The method for factor extraction consisted of the principal factor method using the promax Yamamoto T, Kawaguchi A, Otsuka Y MedEdPublish https://doi.org/10.15694/mep.2019.000091.1 Page | 5 rotation. The Kaiser-Meyer-Olkin (KMO) test and the Bartlett's test of sphericity were used to examine the appropriateness of factor analysis. A result of the former of ≥0.8 and the latter of p<0.05 was considered to define approapriateness.
Step 4: Developing the Level 1 Scale: Main Survey Study design: A cross-sectional study by collective survey using an anonymous, self-administered questionnaire was conducted in September 2016.
Subjects: 131 fourth-year students in the medical department and 40 students in the department of pharmacy of University C located near Tokyo. Survey items: As demographic characteristics, students' academic faculty and department, year, and gender were surveyed. Twelve items were excluded from 35 items used in pre-test, and 12 items were newly added from the item pool, so that 35 items were used in the main survey.
The KiSS-18 is an 18-item scale for measuring social skills in young adults developed in Japan with confirmed reliability and validity (Kikuchi, 2004;Niitsuma et al., 2012), The KiSS-18 evaluates skills in communication and developing personal relationships, and is used to assess criterion-related validity of the present medical professionalism scale. Each question on the KiSS-18 can be answered according to a following five-point Likert scale.
Analysis: Exploratory factor analysis was performed basically the same as that of pre-test. The Pearson correlation coefficient between the KiSS-18 and total score for items that compose the factors were calculated to assess criterion-related validity of the scale. As the KiSS-18 scale assess and measures social skills, the correlations with the medical professionalism scale were as follows: Moderate correlation (0.5) or above was expected with subscale items related to communication skills and interpersonal skills, whereas a mild positive correlation (0.3 or greater but less than 0.5) was expected with other subscale items.
The Cronbach's coefficient alpha for each subscale item and all items were calculated to assess reliability of the scale.

Step 5: Developing the Level 2 scale
Study design: A cross-sectional study of a collective survey using anonymous, self-administered questionnaires was conducted in March -September 2016.
Subjects: Subjects were 352 junior residents immediately before starting their residencies in 35 postgraduate education hospitals affiliated with University B (i.e., immediately after graduation), and 83 fourth-year students in the nursing department of University C. Questionnaires were either collected and mailed back by the person in charge, or were mailed by individual residents.
Survey items: Gender was surveyed as a demographic characteristic. Level 2 items of the medical professionalism assessment scale were selected among the item pool. Fifty items were selected. Twenty-two of the 50 items were reselected among items used in the Level 1 scale. The Japanese version of the P-MEX (Professionalism Mini-Evaluation Exercise) (Tsugawa et al., 2011;Tsugawa et al., 2009), often used as a scale to assess professionalism in residents, were used to test criterion-related validity with the Level 2 scale. The P-MEX is composed of 4 factors and 24 items, among which the second factor, "Reflective skills" (5 items) was used as skills assumed to be largely acquired at the time of graduation. The P-MEX is used to assess others, not for self-assessment, the options for answers were revised to five-point Likert scale.
Analysis: The same methods as those used to for the main survey for the development for the Level 1 scale were used.
SPSS Statistics 22.0 (IBM) was used for the analysis of both surveys, and p<0.05 was set statistically significant.

Ethical considerations
Anonymous, self-administered questionnaires were used for the survey. Subjects were explained orally and in writing of the purpose, methods, ethical considerations and publication of the study results, and the survey was conducted on those who gave consent. Subjects were clearly explained the ethical considerations, including the following: (1)Data that allow identification of individuals would be deleted before use in analysis; (2)Subjects would not be subjected to losses or disadvantages regardless of their decision to cooperate or not cooperate in the study.
This study was approved by the Ethical Board of the Sapporo Medical University (27-2-58).

Level 1 Scale: Pre-test
The response rate was 97.5% (803/824). Respondents were represented in largest numbers of the departments of pharmacy, nursing, medicine, physical therapy and dentistry, in that order. There were more female respondents (60.4%) than males (Table 3).
An exploratory factor analysis was performed to test construct validity. Eleven items that did not meet the criteria presented in the methods chapter were excluded from the factor analysis, and 5 factors composed of 24 items were extracted.

Level 1 Scale: Main survey
The response rate was 90.6% (155/171). There were slightly more male (58.3%) than females (41.7%) ( Table 3). The mean KiSS-18 score ± standard deviation was 62.65±10.44. There are studies that have assessed university students in medical domains to score around 60 points (Fujino et al. 2005;Kudou et al. 2007 ;Yamamoto et al. 2013), the social skills of the subjects of analysis can be estimated to be largely standard.
The Pearson product-moment correlation coefficient between the scores for the 7 factors and the KiSS-18 was 0.23-0.76. The correlation was strongest with Factor 1, "Building interpersonal relationships" (r=0.76), whereas the correlation was weakest with Factor 5, "Knowledge and skills" (r=0.23) and Factor 6, "Ethical and social responsibility" (r=0.23). The correlation coefficient between the total scores for the 30 items and the KiSS-18 was 0.71 (Table 5).
The coefficient alpha of the 7 factors was 0.63-0.86, and the coefficient alpha for all 30 items was 0.90. The response rate was 55.2% (240/435). There were slightly more females (51.7%) compared to males (48.3%) ( Table 3). The score distribution of "Reflective skills" of the P-MEX was 15.33±2.12.
There was a moderate correlation between "Reflective skills," Factor 2 of the P-MEX and the Level 2 medical professionalism assessment scale, at 0.28-0.59 (Table 7). The correlation coefficient with Factor 8, "Reflective practice" for which the details of the assessment items are similar, was particularly high at 0.59.
The coefficient alpha of the Level 2 scale was in the range of 0.71 to 0.87, and the α coefficient for all 31 items was 0.93.  The numbers less than 100 are the items used in common by Level 1 and Level 2 scale. The numbers 100 more than 100 are the items used in only Level 2 scale. The 8 factors were named as follows: "Providing safe, quality care" (Factor 1), "Providing patient-centred care" (Factor 2), "Planned learning" (Factor 3), "Collaborative practice" (Factor 4), "Building interpersonal relationships" (Factor 5), "Interest in community health (Factor 6), "Ethical and social responsibility" (Factor 7) and "Reflective practice" (Factor 8).

Construct validity of the medical professionalism assessment scale
A factor analysis resulted in the extraction of 30 items in 7 factors from the Level 1 scale and of 31 items in 8 factors of the Level 2 scale. The results of Kaiser-Meyer-Olkin test and Bartlett's test of sphericity were good overall.
We will attempt to compare the concepts that compose the ABIM scale (Arnold et al., 1998) and the P-MEX scale (Cruess et al., 2006) to investigate the construct validity of the Level 2 scale. The ABIM scale is composed of the 3 factors of "Altruism," "Respect for others," "Excellence" and "Honour and Integrity." The details of "altruism" and "respect for others" are related to patient-centred care and collaboration with professionals in other healthcare professionals. They correspond to the 2nd factor "Providing patient-centred care" and the 4th factor "Collaborative practice" of our Level 2 scale. "Excellence" also corresponds to the 2nd factor "Providing patient-centred care" and the 4th factor "Collaborative practice." "Honour and integrity" corresponds to the 2nd, 7th and 8th factors, which are respectively, "Providing patient-centred care," "Ethical and social responsibility" and "Reflective practice." The P-MEX (Cruess et al., 2006) has 24 items in the 4 factors of "Doctor-patient relationship skills," "Reflective skills," "Time management" and "Interprofessional relationship skills." Comparing this to Level 2 scales in this study, "Doctor-patient relationship skills" correspond to "Providing patient-centred care" (Factor 2), "Reflective skills" corresponds to "Reflective practice" (Factor 8), "Time management" corresponds to "planned learning" (Factor 3) and "Interprofessional relationship skills" corresponds to "Collaborative practice" and "Building interpersonal relationships" (Factors 4 and 5). As such, the components of the ABIM and P-MEX scales are encompassed by the Level 2 scale of this study, and were also found to be consistent with the concepts of medical professionalism displayed in Table 2. These observations demonstrate that the Level 2 scale is a scale that can comprehensively assess medical professionalism.
Our Level 2 scale contains the concepts of "Providing safe, quality care" (Factor 1) and "Interest in community health" (Factor 6) which are not included in the ABIM and P-MEX scales. Perhaps there is some debate as to whether Factor 6, "Interest in community health," is an element of professionalism that all healthcare professionals should have. However, in recent years, days in hospital are being cut dramatically in acute care hospitals especially in Japan, and with this change, healthcare professionals in acute care are also required to adjust the treatment and care they give to one with consideration for the patient's life and environment following discharge. This suggests that it is very important for "Interest in community health" to be included in the scale for measuring medical professionalism. As such, the Level 2 scale of this study regards the concepts of medical professional more broadly and comprehensively than existing scales of medical professionalism, and is a novel scale that is well adapted to change in the healthcare environment.
On the other hand, for the characteristics of the Level 1 scale 18 items in the field "Personality development and social skills (Category: 11a, 11b, 11c, 11d)" have been selected, which suggests the importance of cultivating professionalism in this field among students before they start their clinical practice. There are no scale items that correspond to the field "Patient-centred care(Category: 21a, 21b, 21c)" in the Level 1 scale, but there are 9 items in the Level 2 scale, which illustrates that this is an aspect of professionalism that is fostered in clinical practice. As such, the scales at the two levels are adapted to the readiness of students, as the Level 1 scale measures professionalism of students before clinical practice and the Level 2 scale measures medical professionalism at graduation.

The criterion-related validity of the medical professionalism assessment scale
To investigate the criterion-related validity of our medical professionalism assessment tools, we examined the correlation between the Level 1 scale and the KiSS-18, which measures social skills. The correlation was the strongest (r=0.76) with the 1st factor, "Building interpersonal relationships," which is the most similar to the details of the KiSS-18, suggesting that the respondents were reacting and answering appropriately to the items and thereby securing criterion-related validity.
We examined the correlation between eight Level 2 subscale items and "Reflective Skills" of the P-MEX. The correlation was the strongest (r=0.59) with the 8th factor of "Reflective practice". This also suggested that the respondents were reacting appropriately to the items, thereby again securing criterion-related validity. The reliability coefficient for the 8 subscales in the Level 2 scale lies in the range of 0.71-0.87, ensuring high reliability. The reliability coefficient α of seven subscales in the Level 1 scale was above 0.6 for all subscale, which indicates that a certain degree of reliability was ensured. However, it might be necessary to reinvestigate the possibility of revising some of the scale items as the reliability coefficient α of three subscale items was below 0.7.

Limitations and future challenges in the medical professionalism assessment scale
The first of the limitations and tasks of this scale, we are yet to verify the cross-cultural validity of the scales. Second, there is room for improvement in the Level 1 scale to increase reliability, and this may require reinvestigation with possible revision of the scale. Third, test-retest reliability should be examined. Forth, this is a self-assessment tool and does not directly measure the medical professionalism abilities acquired by students. The predictive validity should be examined by the longitudinal study.

Conclusion
We developed two medical professionalism assessment scales for students to self-evaluate the levels of medical professionalism that they have acquired. The scales were based on concepts of medical professionalism extracted from various materials on medical professionals published or presented by various professional associations. Factor analysis resulted in 30 items from 7 factors for Level 1 (before clinical practice) and 31 items from 8 factors for Level 2 (at the time of graduation) assessments. Construct validity, criterion-related validity and reliability were generally confirmed, and we will study these issues further to enable their use in evaluation of medical education.

Take Home Messages
The concepts of medical professionalism are changing with changes in the healthcare environment. There are no existing assessment scales of medical professionalism with proven validity and reliability that can be applied for evaluating education. The two new assessment scales developed for medical professionalism can be used at two levels: before starting clinical practice (Level 1) and at the time of graduation (Level 2). The assessment scales warrant further investigations to examine applicability in other cultures and test-retest reliability.

Notes On Contributors
Takeshi Yamamoto is an associate professor of school of health sciences at Sapporo Medical University, Japan. He is a medical sociologist in the field of medical education and medical professionalism. ORCID ID: https://orcid.org/0000-0003-4765-459X Akito Kawaguchi is a professor of school of health sciences at Hokkaido University of Science, Japan. He is a phycisian with research regarding health science.
Yoshinori Otsuka is a professor of faculty of sports & human at Sapporo International University, Japan. He is a phycisian with research regarding health science.