Evolving and evaluating the OMERACT fellows program: insights and implications from OMERACT 2023 fellows

Objective: To describe the evolution of the OMERACT Fellows Program (OM FP) and to evaluate the innovative changes implemented in the 2023 program. Methods: The OM FP, the first of its kind in global rheumatology, was developed in 2000 to mentor early career researchers in methods and processes for reaching evidence-driven consensus for outcome measures in clinical studies. The OM FP has evolved through continuing iterations of face to face and online feedback. Key new features delivered in 2023 included e-learning modules, virtual introductory pre-meetings, increased networking with Patient Research Partners (PRPs), learning opportunities to give and receive personal feedback, ongoing performance feedback during the meeting from Fellow peers, PRPs, senior OMERACTers (members of the OMERACT community) and Emerging Leader mentors, involvement in pitching promotions, two-minute Light-ning Talks in a plenary session and an embedded poster tour. An online survey was distributed after the meeting to evaluate the program.

Objective: To describe the evolution of the OMERACT Fellows Program (OM FP) and to evaluate the innovative changes implemented in the 2023 program.Methods: The OM FP, the first of its kind in global rheumatology, was developed in 2000 to mentor early career researchers in methods and processes for reaching evidence-driven consensus for outcome measures in clinical studies.The OM FP has evolved through continuing iterations of face to face and online feedback.Key new features delivered in 2023 included e-learning modules, virtual introductory pre-meetings, increased networking with Patient Research Partners (PRPs), learning opportunities to give and receive personal feedback, ongoing performance feedback during the meeting from Fellow peers, PRPs, senior OMERACTers (members of the OMERACT community) and Emerging Leader mentors, involvement in pitching promotions, two-minute Lightning Talks in a plenary session and an embedded poster tour.An online survey was distributed after the meeting to evaluate the program.

Background
Outcome Measures in Rheumatology (OMERACT) is a global, volunteer-driven, not for profit organization committed to improving outcomes for patients with autoimmune, rheumatic and musculoskeletal diseases (RMDs) through advancing the design and quality of clinical studies.OMERACT has a distinguished track record of developing and validating core domain sets and core outcome measurement sets (together, core outcome sets (COS)) for use in rheumatology studies, including those for regulatory approval of new treatments.OMERACT has been at the forefront of these efforts for the last three decades.Working groups (WGs) are the core of OMERACT and focus on different areas, including specific MSK conditions, imaging and biomarkers or methodologies [1].OMERACT is seen as an exemplar for its early recognition of the importance of involving early career researchers (Fellows) and Patient Research Partners (PRPs) in all WGs.All OMER-ACT WGs are invited to present the advancement of their work every two years in an international meeting, hosted each time in a different continent.In addition to hosting plenaries with interactive and interdisciplinary breakout groups and special interest group sessions, these meetings provide dedicated time for Fellows to undertake an intensive dedicated program: the OMERACT Fellows Program (OM FP).
The OM FP was conceived in 2000 and has contributed to the sustainability of OMERACT since its introduction.The purpose of the OM FP has been clear since its inception: to provide mentoring and support to instill in a young generation of rheumatology researchers the passion for clinical research based on the development of outcomes proven to be reliable and valid (truthful), able to detect change (discriminant) and to be efficiently implemented (feasible).Over the last 20 years, through ten bi-annual OMERACT meetings, more than 200 young researchers have attended the program, and more than 50 % of them have continued to be involved in OMERACT outcome research.The program has evolved continuously from an informal daily discussion to debrief on the work presented by each group during that day of the meeting, to a more formal methodologically driven education program.Although exhausting, most Fellows describe the OMERACT experience as exhilarating and impactful for their careers.The increasing number of returning Fellows has been seen as a success of the program, leading to the introduction of the Emerging Leaders program in 2016.This program focuses on the development of more advanced skills and competencies in the OMER-ACT process and methods whilst mentoring new Fellows [2].
A review of OMERACT records since the inaugural Fellows meeting in 2000 identified that 208 Fellows, from 16 countries across four continents, had attended an OMERACT meeting.Of these, 107 attended at least one other OMERACT meeting after their fellowship and 81 are still engaged in OMERACT activities in 2023, with 19 having a role with some leadership.Fellows have been involved in approximately 60 different Working Groups covering rheumatology outcome topics encompassing 23 different conditions or health states (e.g.foot and ankle disorders, RA remission, OA Flare, Bechet's, Fibromyalgia), 15 different methods issues (e.g.Quality of life, MCIDs, contextual factors, development of composite indices) and more than 20 different imaging and biomarker groups (e.g.ultrasound, MRI and SpectCT scoring for many joint sites and conditions [RA, OA, JIA and AS], synovial tissue scoring).OMERACT WG and meeting activities have led to over peer-reviewed papers, with the majority of Fellows being able to cite one or more of these publications in CVs, contributing to their career development opportunities.
While the OM FP had been evolving in response to participants' feedback over the years, the pandemic and the accompanying availability and uptake of virtual technologies led OMERACT to undertake substantive changes.The development of e-learning modules that were completed and discussed by Fellows at pre-meeting introductory sessions enabled more time in the face-to-face meeting program for several innovations.The experience of other participant groups at OMERACT meetings has been reported [2,3].This report aims for the first time to report on the Fellows' experiences by providing a brief historical background to the OM FP, evaluating the most recent 2023 program and developing recommendations for future evolutions.

Development of the 2023 program
Formal feedback from the OM FP was introduced following OMER-ACT 2016; overall, the program was well received, with the majority rating the content and opportunity very worthwhile.Recommendations were made for improvements in facilitation of discussion groups generally across the meeting and for providing feedback specifically in the Fellows' presentations in the program.In 2018 OMERACT introduced facilitator workshops and discussions on feedback, and modified the OM FP accordingly.Fellows who attended the OMERACT meeting completed an anonymous online survey after the event.Highlights identified by the 2018 Fellows included connecting with other Fellows and learning from each other, critical appraisal of their work and OMERACT processes, learning from mentors, feeling part of the OMERACT community, and having their opinions valued.Suggestions to improve the 2018 program included the removal of SIG report-backs from the daily summary sessions, shorter days, more organized social sessions, earlier communication about expectations, equal discussions with all stakeholders, an increased focus on methods and the opportunity for all Fellows to present in a plenary session.The OMERACT Fellows Program was developed by a central OMERACT team (including LM, MADA, FG, AK and SG), and led onsite by three OMERACT senior leaders (LM, MADA, FG) who incorporated feedback from the preceding programs as well as input from a pre-meeting survey that was conducted for the first time.

Pre-meeting survey
All Fellows attending OMERACT 2023 were invited to complete an online survey in advance of the meeting to further inform the final program.The survey was distributed through SurveyMonkey and opened three months before the OMERACT 2023 meeting.It stayed open for one month and comprised of multiple-choice questions relating to confidence in core domain set development, instrument selection and consensus methods (rated on a three-point scale from not at all confident to extremely confident), and the importance of OMERACT for networking, feedback on research and career development (rated from not important to very important).The survey also included two openended, free text questions: 'What does a successful OMERACT 2023 look like for you?' and 'What do you want to take away from OMERACT 2023?'.

Program components
Key components of the final OMERACT 2023 Fellows Program are presented in Table 1.In addition to the formal program, optional social activities were added, including a running group and midweek field trip.

Post-meeting survey
Fellows were invited to complete a second online survey, anonymously, following attendance at the OMERACT 2023 meeting.Questions focused on the importance of the program, appropriateness of the workload, the quality and relevance of program components, overall experience and likelihood of future attendance.Full survey details are provided in the supplementary material.

Analysis
Survey data were analysed using frequencies (number (n) and%).Free text comments were analysed by three researchers (SG, AK, LM) using thematic analysis where possible to identify the presence of similar concepts across the dataset and develop common themes.Where there was insufficient data for thematic analysis, free text comments were summarized narratively.

Results
Eighteen Fellows attended OMERACT 2023 in-person.Of these, 17 (94 %) completed the pre-meeting survey and 15 (83 %) completed the post-meeting survey.Survey respondents came from eight different countries (Australia, Canada, Denmark, France, Germany, The Netherlands, United Kingdom, USA).

Confidence in methods
Pre-meeting confidence for core domain set development, instrument selection and consensus methods is presented inFig.2. Overall, respondents were somewhat confident with all methods.

Importance of the OMERACT meeting
Fig. 3 displays the importance of the OMERACT meeting for networking, feedback and career development.Overall, the majority of the participants found that the meeting was very important for networking and to receive feedback on their own research, while the results were mixed for the meeting's importance on career development.

Measures of meeting success
Three themes were identified from respondents' free text responses to the question 'What does a successful OMERACT 2023 look like for you?'.A summary of responses is presented in Table 2.

Anticipated skills
In response to the question, 'What skills do you want to take away from OMERACT 2023?', respondents reported these as networking and communication skills, OMERACT-specific skills and general research skills.Specific examples are presented in Table 3.

Importance of a dedicated fellows program
Based on their experience at OMERACT 2023, 13 out of 15 (87 %) respondents indicated that having an OM FP was extremely important whilst two (13 %) indicated that it was important.

Program workload
The workload associated with the OM FP was deemed slightly above average by eight (53 %) respondents, excessive by six (40 %) respondents, and just right by one (7 %) respondent.

Information and support from the program chairs
Five (33 %) respondents felt the level of information and support for the OM FP prior to and during the OMERACT 2023 meeting was just right, and four (27 %) respondents felt it was slightly above average.The remaining six respondents (40 %) had suggestions about how to do   things differently, including sending Fellows information about the program earlier and reducing the number of emails sent about the meeting.

Quality and relevance of program components
An overview of ratings of the components of the program is presented in Fig. 4. The highest rated components were the PRP/Fellow Connection Carousel and Lightning Talks, whilst the lowest rated component was the COS promoter role.

Pre-meeting 'meet and greet' sessions
The quality and relevance of the pre-meeting 'meet and greet' sessions were rated as exceptional by six (40 %) respondents, good by five (34 %) respondents and average by two (13 %) respondents.The remaining two (13 %) respondents did not provide a rating.Free text comments included the suggestion for clearer instructions in these sessions.

Fellows' introductory session
The introductory session was rated as exceptional by 11 (73 %) respondents and good by four (27 %) respondents.One respondent suggested the session was too theoretical.

Emerging leaders mentoring program
The quality and relevance of the OMERACT Emerging Leaders Mentoring Program was rated as exceptional by five (33 %) respondents, good by six (40 %) respondents, average by two (13 %) respondents and below average by one (7 %) respondent.The remaining respondent (7 %) did not provide a rating.No recommendations for improvement were provided.

Delivering effective feedback session
This session was rated as exceptional by 11 (73 %) respondents and good by four (27 %) respondents.Respondents indicated that they would have liked more preparation time, additional time to practice their Lightning Talks, and a more formal run-through of these talks in this session.

Daily summary session content
The daily summary sessions were rated as exceptional by five (34 %) respondents, good by six (40 %) respondents, average by two (13 %) respondents and below average by two (13 %) respondents.Some respondents indicated that the opportunity to discuss the events of the day was a positive aspect, whereas others perceived these sessions as too informal.Suggestions for improvement included more formal teaching and allocation of daily summary session time to meet with their mentors.

Daily summary session duration
The duration of the daily summary sessions was rated as exceptional by six (40 %) respondents, good by four (27 %) respondents, average by four (27 %) respondents, and below average by one (6 %) respondent.Respondents suggested the 60-minute session was too long and a 30minute session would be more appropriate.

Feedback from program chairs
The chairs' feedback was rated as exceptional by seven (46 %) respondents, good by seven (46 %) respondents, and average by one (8 %) respondent.Free text comments included suggestions for individuallevel rather than group-level feedback and clarification on what happens to Fellows after they have completed the program (e.g. at the OMERACT 2025 meeting).Additionally, one respondent perceived that the chairs were defensive when critiqued.

Time for interactions between fellows
Dedicated time for interactions between Fellows (e.g.networking breaks) was rated as exceptional by eleven (73 %) respondents, good by three (20 %) respondents, and average by one (7 %) respondent.Respondents highlighted that having time in the middle of the day for interactions was a positive aspect.

PRP/Fellow connection carousel
Fourteen (93 %) respondents rated the Connection Carousel as exceptional, whilst one (7 %) respondent rated it as below average.Free text comments included the suggestion to extend the duration of this session and to extend the carousel to include OMERACT veterans.However, one respondent did not like the concept of this component at all.

Plenary lightning talks
Lightning Talks were rated as exceptional by 14 (93 %) respondents, and below average by one (7 %) respondent.Free text comments included the suggestion for concrete examples (e.g. a video demonstration) and for the session to be a competition with the scoring of talks.One respondent felt this session had no value.

Core outcome set (COS) promoter role to create engaging social media posts showcasing key strategies for COS uptake
The quality and relevance of the COS promoter role was rated as exceptional by one (7 %) respondent, good by one (7 %) respondent, average by six (39 %) respondents, below average by three (20 %) respondents, and poor by three (20 %) respondents.The remaining respondent (7 %) did not provide a response.Free text comments indicated the need for more clarity of expectations surrounding the COS promoter role.

Fellows' poster tour
The quality and relevance of the poster tour was rated as exceptional by seven (47 %) respondents and good by six (39 %) respondents.One (7 %) respondent rated the poster tour as below average, and the remaining respondent (7 %) did not answer.Respondents suggested that a more spacious room to allow easier communication during the poster tour and splitting posters by topic would improve the session.

Overall experience and future attendance
On a 1-10 numeric rating scale (1 = I wish I hadn't attended, 10 = it was brilliant), seven (47 %) respondents gave a rating of 10, five (33 %) respondents gave a rating of 9, two (13 %) respondents gave a rating of 8, and one (7 %) respondent gave a rating of 2. Fourteen of the 15 (93 %) respondents indicated that, given the opportunity, they would attend an OMERACT meeting in the future.

Discussion
Our report fully describes and formally evaluates the 2023 OM FP, incorporating feedback from the preceding program.We have identified positive aspects, areas for improvement, and potential solutions for future program development.Ninety-three per cent of the Fellows who participated in the 2023 OM FP had a positive experience and would attend subsequent OMERACT meetings, highlighting the potential of the program for continuity.
Two highly successful components of the OM FP that should be incorporated into future OMERACT meetings were the PRP/Fellow Connection Carousel and the plenary Lightning Talks.Both sessions were new in the 2023 program and reflect the core OMERACT value of open, respectful, and effective communication with all stakeholders, facilitating meaningful engagement and participation.Engagement with patients at OMERACT meetings has significantly influenced the scope and conduct of outcome measurement research in rheumatology [4].Building on the initial success of this program component in 2023, the Connection Carousel could be extended in duration at future meetings.Additionally, the scope could be extended to include 'speed dating' between Fellows and OMERACT veterans/executives, providing an opportunity for equal discussions between attendees at different career levels.The Lightning Talks were recorded; this component of the program could be further enhanced by showing a video demonstration to future Fellows prior to the meeting.
Areas of improvement for the OM FP include a reduction in the workload and clearer expectations, particularly in the pre-meeting 'meet and greet' sessions and in relation to any additional roles for the Fellows within the meeting itself (e.g. the COS promoter role).Specific areas for improvement include the content and duration of the daily summary sessions and the Emerging Leaders Mentoring Program.Based on Fellows' feedback, shorter and more structured daily summary sessions may improve Fellows' experiences of the program.Respondents however did not provide any feedback to guide improvements.Future research could explore Fellows' and Emerging Leaders' experiences of the Emerging Leaders Mentoring Program in more depth, e.g. through individual qualitative interviews or focus groups.To further understand the value and impact of the OM FP, a formal evaluation among the wider OMERACT community could be undertaken, and the transition rate of Fellows to Emerging Leaders and retention in outcomes research could be measured over time.
The 2023 OM FP included representation from eight countries across three continents.However, there was no representation from Asia, Africa or South America, which reflects an ongoing lack of diversity within the OMERACT community.In line with work in progress by the OMERACT Equity, Diversity and Inclusivity Working Group [5,6], a dedicated plenary session was held to address this problem at the 2023 meeting.Future work is needed to ensure maximum inclusion of Fellows from under-represented geographic locations, particularly low-and middle-income countries; access to online teaching materials may improve participation.
A limitation of our report is that one Fellow did not respond to the pre-meeting survey and three did not respond to the post-meeting survey, therefore the data presented may not reflect the views of all Fellows who participated in the program.Additionally, we did not explicitly capture data relating to post-meeting measures of success, confidence with OMERACT methods or perceived importance of OMERACT 2023 for networking, feedback and career development.As such, we were unable to directly compare pre-and post-meeting findings to fully explore the impact of the 2023 meeting.Notwithstanding, this initial insight will contribute to continuing to shape the objectives of the next OMERACT Fellows Program, as well as the content of sessions within the broader biennial meeting.
OMERACT is deeply committed to the development and growth of fellows, recognizing their vital role in shaping the future of Core Outcome Set development.The organization is dedicated to refining and enhancing the Fellows Program ensuring the program continues to foster the fellows' active participation, engagement, and leadership skills.A Strategic Advisory Group (SAG) [7] was formalized at OMER-ACT 2023.A key initiative for the SAG involves establishing well-defined roles and objectives for OMERACT Fellows, fostering clarity and direction in their journey within OMERACT.This effort is geared towards harnessing the full potential of these roles, enriching their experience and impact within OMERACT.
Further, OMERACT advocates for increased involvement of Fellows in meaningful engagements with Patient Research Partners (PRPs) and active participation in key Management activities, including the Technical Advisory Groups (TAGs) and SAG.These collaborations are seen as pivotal in expanding the influence and contributions of Fellows in various facets of the organization Collaborations with past and present Fellows, SAG, TAG, OMERACT Management and Fellows Leadership Team will ensure ongoing evolution of this unique Program.

Declaration of competing interest
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Lara Findings were presented during the final plenary session.Fellows' poster tour In this structured session, conference attendees visited and discussed Fellows' research posters on a guided tour facilitated by the OM FP chairs.At the end of each poster presentation, Fellows were asked questions and scored by PRPs on their ability to convey their work to the audience and the level of patient engagement in the work.

Fig. 4 .
Fig. 4. Overview of ratings for each component of the 2023 OM FP.
(continued on next page) L.S. Chapman et al.

Table 2
Themes relating to meeting success.
Theme SummaryForming new connections and deepening existing relationships Time and events to network and meet others; open in-depth discussions; facilitation of friendships; professional links and connections with colleagues around the world; meeting OMERACT leaders and WG members in person.

Table 3
Summary of anticipated skills.
Chapman Grants or contracts from any entity: HEE/NIHR Clinical Doctoral Research Fellowship (ID NIHR30217) Nilasha Ghosh Consulting fees: Musculo Leadership or fiduciary role in other board, society, committee or advocacy group, paid or unpaid: GoodRx Clinical Review Board Francis Guillemin Grants or contracts from any entity: NOVARTIS Max Weinbrecht-Mischkewitz OMERACT, Minister Erna Hamiltons Legat, and The Parker Institute, Bispebjerg and Frederiksberg Hospital and the Department of Internal Medicine, Rheumatology, Clinical Immunology and Osteology, Schlosspark-Klinik, University Medicine Berlin -Funding for OMERACT 2023 registration fee and travel costs