Characteristics of exceptionally good Doctors—A survey of public adults

Background Systematic reviews have found that doctors can have a substantial effect on patients’ physical health, beyond what can be explained by known factors. In a previous qualitative study, 13 medical doctors were interviewed on their experiences of exceptionally good doctors, and all had met at least one such doctor. Objective To determine how common it is for exceptionally good doctors to be encountered by patients and what are the characteristics of exceptionally good doctors. Design Mixed methods cross-sectional survey of 580 Amazon Mechanical Turk participants. Questions included doctor and participant demographics, and 34 Likert questions on characteristics of exceptionally good and average doctors. Free-text questions allowed participants to describe exceptional doctors, record their experience, and provide survey feedback. Stratified sampling ensured gender parity and 33% of participants aged ≥55 years. Analysis included descriptive statistics, statistical modelling of associations between Likert scale scores and patient demographics, and factor analysis. Results Of 580 responses, 505 (86%) were included in the analysis. Factor analysis confirmed internal validity. Most respondents (86%) had met at least two exceptionally good doctors, of whom 55% were specialists. 58% of respondents regarded doctors as exceptional based on an overall impression with multiple reasons. Doctors were most commonly considered exceptional based on one or more of their personality, diagnostic, or intervention ability. Respondents who reported the doctors “willingly listened to them to the end” scored their doctors higher on 33 of 34 Likert questions, except for popularity. They also rated average doctors lower throughout. Conclusions Exceptionally good doctors appear to be commonly encountered by the adult public. Listening to patients willingly to the end is a highly rated and influential characteristic, suggesting that listening could be targeted for quality improvement.


Introduction
Medical doctors are known to have a clustering effect in clinical trials [1][2][3][4], i.e. patients of a particular doctor tend to have similar outcomes, which is likely due to confounding factors such as differences in patient demographics across practices, but could also be due to doctors having different levels of ability in treating patients.
None of the studies that identified exceptionally good doctors provided recommendations for further research or published further details of such exceptionally good doctors [33][34][35][36][37]. In a recent qualitative study [38], 13 medical doctors stated their thoughts on what are the characteristics of exceptionally good doctors, with being both exceptionally skilled and very good at patient communication considered important. The doctor experiences and definitions from the qualitative study were used to design the present cross-sectional survey of the general public on their opinions on what makes an exceptionally good doctor, and their experiences of such doctors [39]. The survey objective was to determine how commonly exceptionally good doctors are encountered by patients, what are the characteristics of exceptionally good doctors according to patients, whether there are multiple types of exceptionally good doctors, if yes, whether patients evaluate different types of exceptionally good doctors more or less positively and whether patients evaluate exceptionally good doctors differently from average doctors.

Methods
The survey reporting follows the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines [40]. All details are presented in a previously published protocol [39].

Study DESIGN
This is an observational convergent design [41] cross-sectional survey including three qualitative, 19 quantitative (Tables 1-5), and 34 5-point Likert questions (Table 6) where the 34 Likert questions were asked first to characterize the exceptionally good doctor nominated by the respondent and then the 34 Likert questions were asked of the average doctor (no specific doctor). The full survey is in Supplementary Appendix 7.
Participants were recruited through Amazon Mechanical Turk (MTurk) [42]. The MTurk worker population is a suitable proxy for the general population [43][44][45] and has been used extensively by social scientists [46], allowing stratification by gender and age. MTurk workers aged 55 and older and female MTurk workers were oversampled to get to a 50/50 gender split and to have 1/3 of respondents aged over 55. Further details are provided in the protocol paper [39]. Otherwise there were no further exclusions or inclusionsany MTurk worker could participate. Initially all questions were derived from a qualitative study that interviewed 13 medical doctors on their experiences of exceptionally good doctors [38]. The authors conducted a pilot study with 210 respondents and employed a survey specialist with extensive consumer survey experience to improve the quality of the questions. The pilot study showed that respondents understood the term 'exceptionally good doctors' and factor analysis showed that this term was distinct from the term 'doctor'.
The authors investigated alternatives to the term 'doctor', but such attempts were confusing and discouraging for respondents and were not used in the survey. The consent form and questions 5, 8, and 27 clarified that a doctor is a physician by using the term "doctor (physician)".
Ethical approval (#CS03416) was granted by the Bond University Human Research Ethics Committee on April 27, 2022.

Survey sample
Adult MTurk workers were recruited as participants. The sample size was identified based on the results of a pilot study of 210 participants showing 400-450 participants were required to reduce the Likert question margin of error to ~4%. A sample of 580 ensured 500 completed and valid responses.

Data collection
Demographic information collected included the respondents' age in decades; gender; education level; and previous 12 months count of doctor visits. Their IP address (Internet Protocol address) identified their country. Additional questions included the number of exceptionally good doctors and total number of doctors the participants had previously encountered.
The respondents provided details on an exceptionally good doctor, including their estimated age, gender, specialty, and the reason why they nominated that doctor.
The participants were also asked 34 Likert questions, each listing a characteristic, derived from the previous qualitative study [38] rating both the exceptionally good doctor and the average doctor on this characteristic using a scale from 1.0 (completely disagree) to 5.0 (completely agree) ( Table 6). All respondents were asked all 34 Likert questions for both types of doctors rather than random allocation to either Likert questions type to allow within person comparison of exceptional and average doctors.
A subsequent question displayed the subset of Likert questions, if any, where the respondent rated a characteristic of an exceptionally good doctor as 4.5 out of 5 or higher and asked the respondent to nominate the top three of the listed characteristics in order.
Three free-text questions were asked: one at the beginning to nominate 3-5 characteristics of exceptionally good doctors; another  mid-way through the survey to optionally write about their experience of the exceptionally good doctor in their own words, and one at the end of the survey to provide feedback. Bias due to question order was minimized by, where possible, randomizing the order of multi-item questions such as the Likert questions. All quantitative questions were mandatory. Respondents who provided either logically impossible answers or highly uniform answers, i.e. satisficers [47], were excluded.

Statistical analysis
The analysis includes descriptive statistics on the respondents' and doctors' demographics, how the respondents met their nominated doctor, and why they considered that doctor exceptional. The results from the 34 Likert questions for exceptionally good doctors and the average doctors are shown in graphic form as kernel density plots (a smoothed form of histogram) [48]. Factor analysis assessed internal validation of the Likert questions. A linear regression each was run for the mean of the 34 Likert scores for the exceptionally good and the average doctor as factor analysis showed that there were only two factors with Eigenvalues above 1, one for exceptionally good and one for average doctors. Linear regression models were also used to explore the explanatory variables association with the individual Likert scores to identify Likert questions whose associations differed from the other Likert questions. T-values with absolute values ≥ 2.5 (p ≤ 0.01) were used to confirm evidence of association. As answers to Likert questions were not always normally distributed, we also conducted non-parametric and ordered logistic regression. We compared regression results between respondents who were patients of their nominated doctor and those who knew the doctor in other ways. In addition, we  Table 6 List of 34 Likert questions presented to respondents in random order for describing first the exceptionally good doctor they nominated and then the average doctor.
Likert question ranging from 1 (completely disagree) to 5 (completely agree) 1 The doctor cares for patient 2 Acknowledges patient's experience and knowledge 3 Good at following things up or addressing items from prior consultation 4 Listens well, rarely or never interrupts 5 Connects with the patient on a personal level 6 The patient has no fear of the doctor and may see as a friend 7 The patient trusts the doctor 8 He/She sees patient as a whole person not just a collection of symptoms 9 The doctor is very thorough in the patient's assessment 10 The doctor is a very good observer 11 The doctor gives the patient the time needed 12 The doctor is confident 13 The doctor is courageous when making difficult decisions 14 The doctor is good at communicating 15 The doctor is adaptable, i.e. can respond to the unexpected 16 The doctor is honest 17 The doctor is humble 18 The doctor has integrity 19 The doctor is open minded 20 The doctor is organized 21 The doctor is personable 22 Determined to get past bureaucratic obstacles that affect treatment 23 The doctor is understanding and/or shows empathy 24 The doctor avoids using medical terminology I don't understand 25 The doctor is accurate in diagnosing the issue/problem 26 The doctor is good at explaining things 27 The doctor is knowledgeable 28 The doctor is popular (if you have seen the doctor with others) 29 The doctor is in good physical shape 30 The doctor is in good mental shape 31 The doctor is in an especially harmonious or cared for treatment room 32 The doctor is always on time 33 The doctor has patience 34 The doctor is caring C. Schnelle and M.A. Jones Fig. 1. Wordclouds of free-text qualitative questions.

C. Schnelle and M.A. Jones
compared regression results for those who had an outstanding health event and those who didn't.

Factor analysis
Each Likert question is substantially correlated with the other 33 as shown in the factor analysis in supplementary appendix 6, i.e. all 34 measure a similar quality. Factor analysis identified two substantial factors with one constituting being a doctor and the second being an exceptionally good doctor. After varimax rotation both factors were near equal in size with Eigenvalues of 23.1 and 21.1 and all other Eigenvalues 0.78 or smaller. Therefore this survey measures two substantial separate factors only, one on which all Likert questions about exceptionally good doctors load and one on which all for average doctors load. The only question with a negative loading after rotation is "The doctor listens to me willingly to the end" which is negative for the average doctor, implying that listenedto respondents give average doctors lower Likert scores than other respondents.

Respondents' and reported doctors' demographics
Respondents' demographics are presented in Table 1. Of 587 respondents, 505 (86%) completed and provided valid answers. Thirty-five respondents (6%) did not know any exceptionally good doctors. Another 35 respondents were satisficers [49], and 12 respondents did not finish the survey. 53% of respondents were female, 42% were 25-34 years old, 33% were aged over 55 years, 55% had a bachelor's degree, 19% had a masters' degree, and 91% were from the US. 86% of respondents had met at least two exceptionally good doctors in their life. Table 2 provides details on the exceptional doctors with 55% being specialists, 15% cardiologists, 10% surgeons, and 9% emergency physicians. 37% of doctors had an estimated age below 35 years, with 11% aged over 55 years. Most respondents (87%) had visited the exceptionally good doctor in the previous 12 months.

Respondents' perspectives on exceptionally good doctors
No consensus was shown in how many items a doctor had to fulfill to be considered exceptional. Qualities of exceptionally good doctors nominated by respondents are shown in Fig. 1a as a word cloud. Approximately 150 participants quoted verbatim from the highest-ranked google results on exceptionally good doctors and were excluded from the word cloud analysis [50,51]. Participants' experiences with an exceptionally good doctor are summarized in a word cloud (Fig. 1b) and shown as raw data in Supplementary Appendix 2. A total of 468 respondents provided a response and 388 responses included 5 to 673 words.
The Likert question results showed that average doctors were rated with mean scores of 3.5-3.9 out of 5, and exceptionally good doctors at mean scores of 4.0-4.3. Exceptionally good doctors were nominated for three broad reasons: They were exceptional diagnosticians, exceptionally successful with interventions, or exceptionally good at relating to the patient. The respondents gave similar Likert scores to groups of exceptional doctors based on each of these three categories (Table 4, Doctor Evaluations). The exception is listening as outlined below.
The survey respondents were asked to select and rank the three most important questions among the Likert questions they scored 4.5 to 5 (the maximum score). A total of 387 respondents (77%) provided at least one score of 4.5 or higher; with 45 top ranks given to the doctor being knowledgeable, 35 for being accurate at diagnosing, and 22 for communication ( Table 5, Likert ranking).
Results from the linear regression analysis of the mean Likert scores for each respondent showed that respondents aged 55 or older provided higher Likert scores for the exceptionally good doctors (t-value of 4.9, p < 0.001). The 159 respondents who reported "the doctor willingly listens to me to the end" scored their exceptionally good doctor higher than the other respondents (t = 6.9, p < 0.001) but also scored the average doctor more negatively than their peers (t = − 3.3, p = 0.001). Female respondents scored average doctors higher than male respondents (t = 2.3, p = 0.02) and respondents who were patients of the exceptionally good doctor for a long time scored both types of doctors more highly (t = 2.1 and 2.4, p = 0.03 and 0.02). (Supplementary document, appendices 4 and 5). There was no difference in scores between the 334 respondents who were patients and the 218 respondents who knew the doctor in other ways, nor between the 362 respondents who had an outstanding health event and the 190 respondents who didn't. (Not listed).
Appendix 5 shows a summary of the 34 individual regressions for exceptionally good and average doctors to show variations in outcomes for some independent variables. For example t-values for exceptionally good doctor Likert questions for "The doctor willingly listens to the end" are above 2.5 (p = 0.01 or smaller) for 31 of the 34 Likert questions but not for the Likert question "The doctor is popular" .   Fig. 3. Kernel density plots of all 34 Likert questions. Respondents whose doctor listens consider the doctor much more knowledgeable (blue line) than respondents whose doctor does not do that (yellow line). Fig. 2 shows the distributions of answers to the Likert question "The (exceptionally good) doctor is knowledgeable" as histograms and kernel density plots stratified by whether the respondents affirmed that "The doctor listens to me willingly to the end" -blue for "Yes", yellow for "No" [48]. Fig. 3 shows all 34 Likert questions with the question shown in Fig. 2 highlighted. Fig. 3 illustrates where respondents, who were listened to by the doctor to the end, gave higher Likert scores than other respondents with the largest differences for.

Discussion
Of the 580 respondents to our survey of the general adult public, 86% could recall meeting an exceptionally good doctor and describe such a doctor in detail. This result suggests a substantial number of such doctors exist, a finding consistent with a recent qualitative study in which 13 medical doctors were able to recall at least one exceptionally good doctor [38]. Hence, the dearth of research on such doctors or even good doctors is surprising [13][14][15][16][17]53,54].
The survey respondents nominated doctors as being exceptionally good for at least one of three overarching reasons: for exceptional communication with the patient, an exceptional diagnostic, or an exceptionally successful intervention. The respondents gave similar scores to groups of exceptional doctors based on these three characteristics. The respondents therefore echoed medical doctors who stated that a doctor can be an exceptionally good doctor for a heterogenous set of reasons.
Of the variables assessed for association with the Likert scales ratings, three showed consistent positive or negative associations with exceptionally good and average doctors: Female respondents scored all doctors higher than their male counterparts; those aged 55 or higher scored exceptionally good doctors higher but average doctors lower than the younger respondents; and the 154 respondents who, responding to an item in question 17 of the survey, reported the doctor willingly listens to them to the end gave higher scores to the exceptionally good doctor and lower scores to the average doctor.
These 154 listened-to respondents considered their doctor to be particularly knowledgeable, caring, honest, and with integrity in addition to the expected qualities of being understanding, patient, and good at communicating. The quality of listening was associated with a host of seemingly unrelated positive associations for the patients. Patients being more critical of average doctors after meeting a doctor who listens could provide motivation for average doctors to undermine their exceptional colleagues. It is not a surprise that patients want their doctor to listen but there is no published research that shows quantitatively how much more positively listened-to patients rate their doctor. These participants considered the exceptionally good doctor to be substantially more knowledgeable and honest in addition to being better communicators and were substantially less positive about average doctors. These findings need replication but could potentially be a fruitful avenue for further research; addressing questions such as why doctors who listen to the end are considered more knowledgeable and honest, and why are their patients more critical of average doctors?
A medical specialist, in the same qualitative study described the process of listening as: "Every patient, every person, every being is different. Every person has a different reading. So how can you be generalized into a sample or whatever, a random whatever. So that makes exceptional physicians more humble because you can't let go -be complacent. I got books everywhere. … But when I'm with a patient, I'm totally dedicated to listening -by listening I don't know -something comes up, an impress is given, the whole package of treatment comes through -more and more than ever before And that's what [three famous and exceptionally good doctors] did all the time".
These statements suggest 'listening' could lead to more accurate diagnoses and appropriate treatments, supporting the 154 survey respondents' impression.
Currently, there is no definition of what is an exceptionally good doctor, and no characteristic in the survey was nominated as most important by more than 12% of the respondents. This suggests there are multiple ways to be an exceptional doctor. Doctors with exceptional communication qualities, excellent diagnostic abilities, or outstanding treatment success were equally valued by the respondents and the respondents had no consensus on how many qualities are needed to be exceptional. For our previous systematic reviews, we operationally defined an exceptionally good doctor as one who has exceptionally good patient physical health outcomes [5][6][7]. Conversely, our survey respondents took a much broader view on their opinions and experiences of exceptionally good doctors.
This survey has some limitations that need to be acknowledged. First, it was subject to potential non-response bias, as it was unable to know how many MTurk workers accessed the survey but chose not to participate. Nevertheless, response rates are less important than response representativeness [55], which was ensured as our sample of respondents are of gender and age distribution similar to that of the general population of adults. Second, the respondents were English speakers, predominantly from the US. Thus, it is uncertain whether the findings can be generalized to other regions, particularly developing nations. Third, in the US the percentage of health care costs covered by private health insurance (28%) and out of pocket expenses (10%) is higher than in other countries and doctors in the US system may differ from doctors in countries where public health systems pay more than 49% of health care costs [56]. In addition, due to heterogeneity regarding patient demographics, types of interventions, and types of outcomes relevant to different medical conditions, there may be differing criteria on what makes an exceptionally good doctor for different medical/surgical specialties and doctors and patients may differ in their perceptions of what makes an exceptionally good doctor as their perceptions differ in areas such as acute pain [57]. Despite the limitations, this survey of adult public provides an insightful view of exceptionally good doctors, who appear to be commonly encountered by the general adult public. They tend to be exceptional communicators, diagnosticians, or interventionists. The highest ratings for exceptional doctors are given by patients whose doctors listen to them willingly to the end. The ability to attentively listen makes an exceptionally good doctor stand out among their peers but its lack then also makes average doctors appear worse. Targeting listening skills for quality improvement could improve patient perceptions of doctors and potentially lead to better patient outcomes and higher doctor satisfaction.

Author contribution statement
Christoph Schnelle: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.
Mark Jones: Conceived and designed the experiments; Analyzed and interpreted the data; Wrote the paper.

Funding statement
This review has been funded by the first author as part of his PhD studies. No external funding was received.

Data availability statement
Data will be made available on request.

Declaration of interest's statement
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information
Supplementary content related to this article has been published online at [URL].