Research culture in Australian and New Zealand radiation oncology: Fact or fantasy?

Fostering a research culture is a key goal of the Royal Australian and New Zealand College of Radiologists, yet there has never been an organization‐wide enquiry into the extent to which this is being realized. The purpose of this work was to address that deficit for the Radiation Oncology (RO) Faculty to serve as a baseline for future comparison. The hypothesis was that such a culture is closer to fact than fantasy.


Introduction
One of the strategic priorities of The Royal Australian and New Zealand College of Radiologists (RANZCR, the College) is clinical excellence, with fostering a research culture in medical imaging and cancer care as a key goal therein. 1 This is formalized in a Research Action Plan which is updated from time to time. The most recent plan proved to be unsatisfactory due to broadly defined goals and outcomes difficult to measure. Accordingly, it underwent re-drafting in 2022, aiming to generate a small number of prioritized, well-defined, achievable and measurable goals (formal ratification pending at the time of writing). This proved to be a challenging task for several reasons, one of which is that the research programs of the two College Faculties, Clinical Radiology and Radiation Oncology (FRO), are at different stages of evolution.
It is self-evident that, to determine the ultimate success of the new Research Action Plan, an accurate picture is needed of the baseline extent to which Fellows currently embrace research. FRO promotes research in several ways. The training curriculum includes mandatory participation in at least one of two Faculty run Research Workshops and completion of a research project to qualify for specialist registration. 2 It also allocates approximately AUD$100,000 of seed funding and prizes per year for trainees and junior Fellows to conduct research. 3,4 However, there has until now been no formal review of overall research activities at the national level for either Faculty. Indeed, to the best of my knowledge, nor has there been for any similar societies overseas.
The College's Continuing Professional Development (CPD) program, in compliance with the requirements of the Medical Board of Australia and the Medical Council of New Zealand, facilitates self-directed, self-reported recording of relevant activities, a necessary requirement for maintenance of specialist registration. 5 It is under the supervision of the College's Professional Practice Committees, claimable items and scoring determined periodically by consensus, including feedback from the College membership. This program also underwent a major revision for 2022 after earlier changes in preparation for reforms in regulatory requirements from July 2022 in NZ and January 2023 in Australia. The activities for RO prior to this were grouped into 7 broad categories, namely, (1) Professional and Clinical Governance, (2) Teaching, Training and Supervision, (3) Self-Directed Learning, (4) Research, (5) Publications and Presentations, (6) Attendance at Conferences and Meetings, and (7) Maintenance of Professional Standards. CPD points were claimed each year based upon the number of activities (for example, 20 points for a plenary/keynote speech) or, for some categories, the time spent (for example, 2 points per hour for a grant application review). The allocation generally reflected relative time and effort required, biased towards active rather than passive participation. Compliance was dependent upon accruing specified points targets. For RO, this was a minimum of 50 points per year and 180 points per triennium, in a minimum of any 3 of the 7 CPD categories over the triennium. This was modified in May 2020 to a minimum of 50 points for 2019 only and 130 points for the 2019-2021 triennium due to the impact of COVID-19. The program is subject to annual auditing by the College (7% random sample of members per year) whereby prescribed substantiation of CPD activities is required (for example, electronic citation of a claimed journal article), although auditing was temporarily suspended in 2020, again due to COVID-19. The Medical Board/Council in each country also regularly audit medical practitioners randomly.
Importantly, in terms of interpreting the results of this work, whilst research is strongly encouraged by the College, it is not mandated post-Fellowship. Nor is claiming of CPD research activities, or contemporaneous uploading of substantiation documents (which can instead be provided in the event of auditing).
The purpose of this project was to assess, for future comparison, the current research output of the (approximately) 500 Australian and New Zealand (ANZ) ROs by interrogating the College's CPD database. This approach was anticipated to achieve a much more comprehensive overview than a conventional member survey, given the almost universal poor response rate of such surveys and the likelihood of responder bias towards active researchers. It would also be much more comprehensive (and feasible) than an internet search of research activities using publicly available RO departmental staff lists, because r esum es and curriculum vitae are not accessible for all individuals and, in any event, these documents typically omit many relevant research-related activities. Similarly, although organizations like the Trans Tasman Radiation Oncology Group (TROG) conduct substantial RO research in ANZ, by no means all ROs participate (there are about 160 TROG Full/Life members) and much investigator-initiated research occurs independently of TROG. In any event, their records do not capture anywhere near the detail available in the RANZCR CPD database.
The hypothesis was that a culture of research among ANZ ROs is closer to fact than fantasy, acknowledging that, in the apparent absence of similar comprehensive data from elsewhere, this is a subjective call. Note that the corresponding enquiry for the Clinical Radiology Faculty (which has approximately six times the number of RANZCR members as FRO) was beyond the scope of this work.

Methods
Permission to undertake the project was granted by the College secretariat and RO Dean in June 2022. Because no individual RO (or patient) data were identifiable, Research Ethics Committee approval was deemed unnecessary.
To obtain a contemporary picture of research engagement, it was elected to focus upon the 2019-21 triennium, accepting that COVID-19 from early 2020 inhibited subsequent research output. It was anticipated that 2019 would be representative of pre-pandemic levels, and that 2020-21 could serve as internal validators of those outcomes, albeit suppressed in magnitude. The relevant CPD Categories were Research (Category 4) and Publications and Presentations (Category 5), within which there were 25 sub-categories covering a broad range of activities including grants, research projects, patient recruitment, quality assurance, peer review, conference/meeting presentations and publications. 5 Data were provided by the College as three Microsoft Excel spreadsheets in June 2022. Individuals were anonymized by assigning each a random ID number which applied across any CPD activities they recorded during the triennium.
The primary endpoints were the numbers and percentages of ROs claiming at least one research-related CPD activity overall and in the various CPD sub-categories, by year. The denominators for each year were defined as the total number of registered FRO Fellows (including Life Members), and Educational Affiliates (overseas trained ROs registered to practice but not College Fellows) less the small number of Fellows practicing overseas, mainly in Singapore, who take up the option of instead participating in the CPD program of their local jurisdiction (typically about a dozen yearly). From College records, the denominators for 2019-21 were determined to be 482, 496, and 511 individuals, respectively. Secondary endpoints were "breadth" and "depth" of research undertaken. As a measure of breadth, the distributions of the numbers of different research-related sub-categories claimed by ROs were determined by year. As a measure of depth, the numbers and percentages of individuals who only claimed activities which may be considered "lower-level" were also calculated by year. The sub-categories selected for this measure were manuscript review, trial patient recruitment, trial quality assurance, and conference session chair/panel member, each of which attracted the minimum allocated CPD score of 1 point per unit. Note that research sabbatical (scoring 1 point per hour) was not included here because this would not ordinarily be considered a lower-level activity. By way of comparison, major undertakings like first author or clinical trial principal investigator scored 15 and 30 points per unit, respectively.
Unfortunately, because of the above de-identification process and the substantial time it would have taken College staff to sub-categorize the data for the purposes of this project, subset comparisons for Life Members, Educational Affiliates, and by gender, age, Australia vs NZ, public vs private (or mixed) practice, rural vs metropolitan etc., were not possible.
Before undertaking the analysis, the CPD data were scrutinized for potential errors. Quite a few clearly implausible claims were noted and corrected. Given that the online algorithm automatically calculates points for each recorded activity, most of these were attributable to erroneous input of anticipated points rather than activity units (for example, the claim "editor of 10 journals" scoring 100 points was obviously intended to be "editor of 1 journal" worth 10 points). The remaining errors likely reflected inadvertent double keystrokes (for example 23 units instead of 2), or random entry mistakes. In some cases, substantiation documentation (only available to College staff, to preserve anonymity), enabled resolution of queries by phone. For the handful of cases where there was persisting doubt, high outlier claims were replaced with more plausible, conservative (lower) values, taking into account the overall activity distribution for that sub-category and the claims profile of the individual in question. In no cases were claims upgraded. The aim was to neither falsely inflate the outcomes nor dismiss genuine outlier claims made by exceptionally productive researchers.

Results
The primary outcomes are summarized in Table 1. The overall numbers and percentages of ROs claiming at least one research-related activity for years 2019-21 were 341 (71%), 220 (44%), and 319 (62%), respectively. Reflecting the impact of COVID-19, it will be seen that items relating to conferences and meetings were especially affected. For this reason, it was inappropriate to aggregate the results across the triennium.
Members claimed activities in 23 of the 25 eligible subcategories (there were evidently no editors or first authors of research books during this period). 5 The numbers and percentages of ROs claiming at least one item from each of these 23 sub-categories are displayed in descending order for 2019 (the most representative year). The COVID-19 effect is again evident. The commonest activity in each year was co-author of journal article(s), 25%, 16%, and 27% of members, respectively. The 10 commonest claims for each year were comprised of the same sub-categories, although the order differed. The remaining 13 sub-categories attracted claims by <5% of members each year. For the individuals reporting in each subcategory, the distributions of the numbers of units claimed were markedly "left-skewed" as might be expected, the vast majority of medians being 1-2 units per individual with little variation by year. For the 7 items scored by number of hours spent, the medians were typically 2-5 h per individual, again with little variation by year. However, the majority of the ranges were broad, reflecting high output from some research focussed (academic) individuals.
The secondary outcomes, breadth and depth of research, are summarized in Tables 2 and 3, respectively. The distributions of the numbers of different subcategories claimed by all individuals per year were again "left-skewed", with a median of 1 for 2019 and 2021 (0 for 2020), but a range 0-10 per individual for each of the 3 years. When restricted to the members who submitted at least one research-related CPD claim, the corresponding figures were 2 (range 1-10) each year ( Table 1). The percentages of all individuals only making claims in one of the four lower-level (1 point per unit) sub-categories ranged between 4.4% and 5.9% per year.

Discussion
This project assessed the extent to which ANZ ROs currently embrace the research culture espoused by their College, utilizing data from its compulsory CPD program. It found a significant level of engagement in research as self-reported by members in the latest triennium. Data from 2019, which are most representative of the true contemporary picture, revealed that 71% of ROs claimed at least one research-related activity, with noteworthy percentages publishing (25% co-authors, 10% first authors), actively participating in conferences (15% invited lecturers, 12% oral or poster presenters, 5% session chair/panel members, 4% keynote speakers), leading clinical trials (14% principal, 11% co-investigators), and peer reviewing manuscripts (14%). Other common activities included formal in-house/local meeting presentations (17%) and patient recruitment to clinical trials (10%). For those reporting research-related activities, the median number of sub-categories claimed, namely 2 (range 1-10) remained constant throughout the triennium. Importantly, only about 5% of ROs overall solely The rows are in descending order for 2019 (with highlighting of the percentages for clarity), the year most representative of true contemporary research activity levels. †Hours spent on the activity. All other corresponding entries refer to activity units (for example, the number of co-authorships).  claimed one of the four lower-level activities (see above) each year. Not unexpectedly, there was an obvious drop-off in research activity due to COVID-19 in 2020 (44% claiming at least one sub-category), with significant recovery in 2021 (62%), although not to pre-pandemic levels.
As mentioned, it was not mandatory during the study period to report research in the CPD program (none of the minimum 3 of 7 categories were specified). Indeed, many individuals could satisfy the points requirements without including any research and, additionally, there is a disincentive to comprehensive reporting in that it creates an extra burden of substantiation in the event of auditing. For these reasons, the above results are almost certainly an under-representation of the true research engagement of ANZ ROs.
In the apparent absence of comparable comprehensive studies locally or overseas, it is difficult to put these findings into perspective. There are numerous publications in the literature on medical research activity, but they are typically limited in scope (e.g., publications/presentations and/or projects only 6-10 ), representativeness (e.g., academic centres only 6 ), breadth (e.g., medical residents only 7,9 ), geography (e.g., one health district only 8 ), selectivity (e.g., mixture of specialties 8 ), and/or granular detail (e.g., first vs co-authorships 6,8 ). Similarly, annual reports from individual centres and specialist colleges also lack relevant detail, listing research projects and conference presentations/publications without reporting the activity percentages or distributions among the membership. Of note, based upon publications (alone) normalized per head of population, Berger et al. demonstrated in 2021 that ANZ ROs have been the most productive by geographic region in the last decade, accounting for an approximately 40% contribution to the total, followed by North America, Europe, Asia, South America and Africa. Although these data were actually reported (graphically) for "Oceania" rather than ANZ, the output from the other island nations of Melanesia, Micronesia and Polynesia comprising Oceania is negligible. 10 This certainly lends support to the proposition that a research culture is well developed in ANZ ROs. However, whilst journal publications may be considered the ultimate measure of research productivity or achievement, they rely upon a host of other associated activities like patient recruitment, quality assurance and peer review which are captured in FRO's CPD sub-categories and should ideally be included in any comprehensive assessment of research engagement.
It is important to note that participation by 100% of Fellows is unrealistic, given that research cannot be mandated and given the frustrating reality that it is often conducted with limited (or no) protected time, typically in-between or during clinical duties and/or after-hours. Funding and regulatory challenges add to the burden. Further, the large cohort working solely, or partly, in private practice obviously have competing pecuniary priorities, although there are many ROs who, admirably, are able to combine both in ANZ. Research success also requires an underlying interest in contributing to the literature, dogged persistence and attention to detail, attributes not possessed (or aspired to) by all. In view of these factors, although we anticipated considerable engagement because of the longstanding research focus of FRO, the creditable findings presented here were quite surprising to me and several other experienced colleagues who significantly under-predicted the activity percentages actually observed.
It is intended that this overview will be repeated in several years' time as part of the evaluation of the new Research Action Plan. Although the CPD program has also undergone a major revision, exactly the same claimable items in Categories 4 and 5 are being carried forward, rendering viable a direct comparison of the primary endpoints reported here. There will be some loss of detail for the secondary endpoints because units and points will be replaced by hours spent for all subcategories. For example, the number of publications will be retrievable from substantiation documentation rather than recorded directly, and so will likely be subject to sampling only. However, the CPD program continually evolves in response to feedback and need, and this issue may well be addressed in due course. This work has several limitations. First, self-reporting obviously raises the possibility of data fabrication, but the random auditing undertaken by the College and external authorities in both countries serves as a strong disincentive to such unethical behaviour. Although suspension or termination of College membership for ROs can, in principle, be invoked for failure to respond to requests for information or non-compliance, this has never proven to be necessary. Indeed, the vast majority of apparent claim discrepancies have, on enquiry, been found to be due to unintentional coding errors, and the percentage of valid claims on audit is typically 98-99%. Accordingly, auditing is largely an educational, rather than punitive, process (personal written communication with RANZCR Project Officer). Second, although most research sub-categories are unambiguous, one or two are subject to interpretation. For example, "small group practical instructor at a meeting or course" or "formal presentation at an in-house or local meeting" could mean teaching established, basic skills and knowledge rather than presenting new techniques or research findings. Substantiation documentation typically does not enable this degree of discrimination, although some of these non-research related activities will have been correctly claimed instead in CPD Category 2 (Teaching, Training, Supervision). 2 Third, only one year (2019) truly reflected contemporary research engagement. Reassuringly however, the overall findings were otherwise very similar in the following two COVID suppressed years. Fourth, it was not possible to analyse sub-sets of the CPD data, although small numbers would have limited reliability for two of these in any event (Educational Affiliates and Life Members each constitute only 1-2% of the membership). Fifth, some judgement was required to correct obvious coding errors in the CPD database, but all amendments were conservative (lower scores). Lastly, the lack of suitable comparator studies limited the ability to objectively test the stated hypothesis.
In conclusion, the data presented here lead me to contend that the hypothesis of this project may be correct. Although still a work in progress, a culture of research in RO is arguably more fact than fantasy in ANZ and it is likely that promotion of research by FRO has contributed substantively. This work will serve as a baseline for future evaluation of the new Research Action Plan. It is hoped that it may also stimulate comprehensive assessment of research engagement in other healthcare settings.