Pilot study assessing reflection ability of practicing Australian and New Zealand ophthalmologists using the Groningen Reflection Ability Scale

Background: The benefits of reflective practice to foster high standards of patient care are well documented. Reflective practice is a continuing professional development (CPD) requirement of the Royal Australian and New Zealand College of Ophthalmologists (RANZCO), the Australian Health Practitioners Regulation Agency (AHPRA) and the Medical Council of New Zealand (MCNZ). However, the level of reflection ability among Australian and New Zealand ophthalmologists has not been previously measured. Methods: Fellows of RANZCO were surveyed in late 2011, 18 months after reflective practice became an AHPRA requirement for medical registration. Fellows undertook the college regular triennial CPD feedback survey and were given the option to proceed to the Groningen Reflection Ability Scale (GRAS, English version) test questions using a 5-point Likert scale (1=strongly disagree, 5=strongly agree). Results were collated using SurveyMonkey®. Average response for each question was calculated, and a single score calculated for the responding ophthalmologists. Results: One hundred and sixty Fellows participated in the survey, 89% Australian and 11% from New Zealand, and 81% male. Fifteen percent of practicing ophthalmologists responded. On average, responding ophthalmologists showed a high level of personal reflection ability, with total score 90.78 (possible range 23 – 115). Small numbers, potential bias among responders and lack of a control group limit the study. Conclusions: RANZCO ophthalmologists responding to the GRAS instrument in 2011 on average demonstrated a high level of personal reflection ability. This study is a useful baseline for future studies of ophthalmologists’ reflective practice and of educational interventions designed to improve reflective behavior.


Introduction
Reflection and reflective practice are recorded frequently in the literature, and thought to represent an important skill in sustaining careers in medical practice. The concept of reflection was developed by Dewey (1933). Argyris and Schön (1974), Schön (1983) and Epstein (1999) progressed the conceptualization of reflection in medical practice. There are many models of reflection, all include iterative dimensions (the process of reflection is triggered by experience) or vertical dimensions (different, deeper levels of reflection on experience, reviewed in Mann et al. 2009, Ménard andRatnapalan 2013); correspondingly there are many definitions of reflection. One useful definition is 'a metacognitive process that occurs before, during and after situations with the purpose of developing greater understanding of both the self and the situation so that future encounters with the situation are informed from previous encounters' (Sandars 2009). The various models also describe varying purposes of reflection, which can include reflecting to improve judgements and, personally develop and contextualise practice (reviewed by Chaffey et al. 2012).
Reflection is thought to be important in medical education (Mann et al. 2009, Sanders 2009, Koole et al. 2012) and medical practice, including accuracy of diagnoses (Mamede et al. 2008), and patient safety (Sibinga and Wu 2010). Developing a better understanding of one's individual reflection profile may assist in learning self-regulation (Sobral, 2001). Reflection has been proposed to be important in revalidation of medical licensure (Murphy et al. 2012, Wittich et al. 2014, and adopted worldwide by various professional bodies and medical regulators in continuing professional development and revalidation activities. Mandated professional development plans including demonstration of the learning cycle as a product of reflection can work as an appropriate vehicle for CPD (Institute of Medicine, 2010).
Since 2009 the RANZCO CPD program has required annual demonstration of reflective practice, usually in the form of a clinical audit. The Medical Board of Australia (MBA) requires 'practice-based reflective elements' in CPD programs, and the MCNZ requires practicing doctors to 'demonstrate reflectiveness.' It is important to note potentially negative effects of reflection exist and include reflection without learning and 'recipe-following' (Boud andWalker 1998, Boud 1999). The value of reflection has led to development of portfolios and other tools to inspire and document reflection.
Many instruments have been designed to measure reflective thinking (for example the Cognitive Reflection Test by Frederick, 2005), and reflection in medical students, including the Reflection Questionnaire (Kember et al. 2000), Reflection in Learning Scale (Sobral 2001), Self-Reflection and Insight Scale (Grant et al. 2002), and the Groningen Reflection Ability Scale (GRAS, Aukes et al. 2007). As reviewed by Kalk et al. (2014) each has a different theoretical basis. The GRAS is based on the Five Factor personality theory (reviewed by Digman 1990) and educational practice, and is a one-dimensional scale which covers three dimensions of reflection: self-reflection, empathetic reflection and reflective communication. GRAS has reliability (Cronbach's alpha) 0.83 (1 st measurement) and 0.74 (2 nd measurement).
There is scant data on reflection in practicing doctors. In a systematic review Mann et al. (2009) found that reflection by practicing physicians has been demonstrated, but does not occur in all situations, and is stimulated most often by complex clinical problems. No studies have addressed outcomes of reflective practice beyond self-reporting. There is no published data of reflection by ophthalmologists. We report here the first measurement of reflection in practicing ophthalmologists using the GRAS instrument.

Participants
The study was conducted among ophthalmologists in Australia and New Zealand who were Fellows of RANZCO, a membership-based body responsible for the training and professional development of ophthalmologists in Australia and New Zealand (www.ranzco.edu). Over 98% of practicing ophthalmologists are society members.

Ethics
Ethics approval was granted by the RANZCO Human Research Ethics Committee as a low or negligible risk study.
The study complied with the Declaration of Helsinki. Participants cannot be identified from the responses. Participation was voluntary.

Instrument
GRAS (Aukes et al. 2007) English language version was used (Table 1). This consists of 23 items measured on 5point Likert scales with scores ranging from totally disagree (1) to totally agree (5). Five items (14,17,18,22,23) are differently worded or negated, so that they should be reversed when scored. Individual scores can be summed to a total GRAS score ranging from 23-115. GRAS was administered as Part B immediately following the 22 item RANZCO triennial survey seeking user feedback on the CPD program (Part A, listed in Table 2), which included a number of questions which could be considered to assess reflection. Participants were not told the nature of the GRAS instrument and the word 'reflection' was not mentioned. Participants were offered a bottle of wine for completing the surveys.

Survey administration
We used the web-based instrument distribution service SurveyMonkey® to collect data during December 2011 to March 2012. Fellows were invited to participate in the survey through the RANZCO on-line weekly communication on several occasions and notification on the RANZCO website. As respondents were not identifiable, it was not possible to send reminders to non-responders.

Response analysis
Not every participant responded to every test item. A complete data set would have comprised responses to all 23 test items by all 160 participants. For demographic data responses for each test item ranged between 147 and 157, with the remainder of the 160 participants skipping the question. For each test item responses ranged between 120 and 124 responses. We assumed that the skipped answers would not have significantly altered the data set, and that the actual responses represented the group as a whole. Range of responses was examined for normal distribution. We further assumed an interval level of the data, i.e. that the difference between scores of the Likert-scale is equal along the scale, based on normal distribution of scores found in previous studies. For each test question, we calculated an arithmetic mean score from the number of responses at each Likert score. The mean scores were then summed to give a total GRAS score for the group. As scores were aggregated in SurveyMonkey®, scores for individuals and subgroup analysis was not possible.

Statistical analysis
Demographics of survey respondents were compared with the population of RANZCO ophthalmologists using Pearson's chi-squared test for association. P value less then 0.05 was considered significant.

Results
One hundred and sixty participants completed the survey in whole or part, of which 80.7% were male, 94.3% in active ophthalmic practice, and 69.3% practicing in metropolitan centres in Australia. Our sample represents 15% of practicing ophthalmologists. There was no significant difference between respondents and the college members as a whole on available data (Table 3).
Responses to all test questions using a five-point Likert scale were normally distributed (data not shown).
On average responding ophthalmologists had a mean GRAS score 90.78, in a range from 23 to 115 points.

Discussion
Personal reflective ability and reflective practice is critical in practicing physicians in generating improved practice performance and patient outcomes. This study investigated the reflective ability of ophthalmologists from Australia and New Zealand using the GRAS instrument. The respondent population was comparable to RANZCO ophthalmologists. On average responding ophthalmologists presented a high score, suggesting a high level of reflective ability. The normal distribution of responses to all test questions is consistent with most ophthalmologists having a high score, rather than responses at the extremes of the scale averaging to a high level of reflective ability. The result is similar to findings in advanced medical students (Aukes et al. 2007(Aukes et al. , 2008. This information is a useful baseline taken close to the time of initiation of requirement for practice-based reflective elements by the MBA. It is not possible to comment on reflection in ophthalmologists compared to other medical specialities, nor on change in ability to reflect during the transition from medical student to trainee to practicing ophthalmologist. The GRAS instrument is a one-dimensional scale designed to measure reflection by medical students, and it has been validated for this use in its original Dutch. Its developers state that it is able to measure self-reflection, empathetic reflection and reflective communication. However, a recent validation study of a Danish version by Andersen et al. (2014) was unable to replicate these findings, and further research was recommended. Accordingly, we have reported a unidimensional result only.
The GRAS is self-rated and this may limit its use, as this presupposes prior self-reflection and self-observation. Research has demonstrated that it is difficult to self-assess performance, with both over-estimations and underestimations possible (Gordon 1991, Woolliscroft et al. 1993, Kruger & Dunning 1999, Eva & Regehr 2005, Davis et al. 2006; this is used to justify the need for external revalidation of medical licensure.
The GRAS instrument was administered directly after the RANZCO CPD survey, without any introductory statements describing the purpose of the questions. Respondents may not have been grounded in context, and this may have influenced responses. Questions relating to self-criticism and awareness of other cultures may reflect individual psychodynamics and cultural awareness rather than reflective ability.
The response rate was 15%, which is comparable to average response rate for electronic questionnaires (Cook et al. 2000). Attempts were made to increase the response rate using RANZCO electronic newsletter reminders, but there were no personal contacts made or CPD points offered for survey completion, both of which may have improved the SurveyMonkey® was used to collate data as it allowed internet data collection and is available at no cost. Crowdsourcing of data is increasingly used in medical research (reviewed by Ranard et al. 2014 andWang et al. 2016). This limited the study since an overall score for ophthalmologists was calculated, but not scores for individuals. SurveyMonkey® also allowed respondents to skip questions, and response rate was lower in the last questions.
Strengths of our study are our demonstration using the GRAS instrument that respondent ophthalmologists had a high degree of reflection, similar to advanced medical students. Responders were representative of RANZCO ophthalmologists across Australia and New Zealand. The information is a useful baseline at a time when the MBA instituted a requirement for reflective practice. Undertaking the survey with no funding was feasible, and may be useful in cultural change promoting the value of reflection by practicing ophthalmologists.
Weaknesses of the study include the low response rate. Using SurveyMonkey® resulted in the inability to calculate GRAS score for individuals, only for the group as a whole. The method of de-identification (collation of on-line anonymous results) will not allow for retesting individuals over time to demonstrate change in reflective ability for individuals. The GRAS instrument basis on personality is problematic, it has not been validated in practicing physicians and low scores have not been associated with poor performance in practice. It has not been widely used in education research; further studies are necessary.
We recommend researchers using instruments to measure reflection in practicing physicians to carefully review available tests, to understand the psychological theory underlying their development and the elements of reflection which they assess. Test instruments will need to be validated in practicing physicians (as opposed to medical students), have reliable test-retest data, and to have scores related to performance in practice. Test methodology will need to force answers to all questions (to ensure a complete data set), allow calculation of scores for individuals, and allow for de-identified retesting of individual physicians over time.

Conclusions
The reflection ability of practicing ophthalmologists in Australia and New Zealand was assessed using the GRAS test instrument. Measuring the ability of reflection has potential use in assessing reflection at both an individual and a group level, in evaluating the effects of teaching, curriculum and revalidation of medical licensure. GRAS is simple to administer. Further research is needed in its validation, particularly in practicing physicians. Further studies will require refinement of data collection to allow calculation of individual scores and follow-up over time of individual responses.  Table 2 RANZCO CPD survey questions responded to by ophthalmologists in Australia and New Zealand using a 5-point Likert scale. This survey was presented before the GRAS instrument (Table 1). Between 144 and 152 ophthalmologists responded to each question.

Table 3
Demographics of survey respondents compared to the RANZCO population of ophthalmologists in Australia and New Zealand  Doing CPD activities in the professional values category has improved my patient outcomes.
10 I am confident that doing CPD improves the quality of my work.
11 CPD is simply a 'numbers game' it is not changing my practice.
12 I understand the importance of doing a surgical audit.
13 I know how to do a surgical audit.
14 The Guide to Surgical Audit and Peer Review is useful.
15 The Guide to Surgical Audit and Peer Review needs revision.
16 Doing a surgical audit improves my patient outcomes.
17 RANZCO needs a binational audit of cataract and other surgical outcomes (similar to RCOphth).
18 It is useful to write a CPD personal development plan.
19 It is difficult to find appropriate CPD activities for each category.
20 I attend CPD activities as they present themselves without much planning of how they align with gaps in my knowledge.
21 Reflection on CPD activities after the event, and claiming points for doing so, is useful.

Take Home Messages
The reflection ability of practicing ophthalmologists in Australia and New Zealand was assessed using the GRAS test instrument.
Measuring the ability of reflection has potential use in assessing reflection at both an individual and a group level, in evaluating the effects of teaching, curriculum and revalidation of medical licensure.
Further studies will require refinement of data collection to allow calculation of individual scores and follow-up over time of individual responses.