Gamification to Engage Clinicians in Registering Data: A Randomized Trial

Objective: To determine the effect of additional gamification elements in a web-based registry system in terms of engagement and involvement to register outcome data, and to determine if gamification elements have any effect on clinical outcomes. Methods: Randomized controlled trial for gynecologists to register their performed laparoscopic hysterectomies (LH) in an online application. Gynecologists were randomized for two types of registries. Both groups received access to the online application; after registering a procedure, direct individual feedback on surgical outcomes was provided by showing three proficiency graphs. In the intervention group, additionally gamification elements were shown. These gamification elements consisted of points and achievements that could be earned and insight in monthly collective scores. All gamification elements were based on positive enforcement. Results: A total of 71 gynecologists were randomized and entered a total of 1833 LH procedures. No significant difference was found between the groups in terms of engagement and involvement on a 5-point Likert scale, respectively 2.34±0.87 versus 2.56±1.05 and 3.63±0.57 versus 3.33±1.03 for the intervention versus the control group (p>0.05). The intervention group showed longer operative time than the control group (108±42 vs. 101±34 minutes, p=0.04), no other differences were found in terms of surgical outcomes. Conclusions: The addition of gamification elements in a registry system did not enhance the engagement and involvement of clinicians to register their clinical data. Based on our results, we advise that registry systems for clinical data should be as simple as possible with the focus on the main goal of the registry.


Introduction
The administrative responsibilities of clinicians are currently much to complain about and the struggle to balance this "paperwork" burden with the clinical care is cumbersome. Clinicians are increasingly imposed to register a wide range of data that is intended to use for quality assessment. It is shown that on average a doctor spends around 17% of working hours on his/her administrative responsibilities only, and this percentage is even increasing (Woolhandler & Himmelstein, 2014).
However, the use of registered clinical data for auditing is recognized as an important tool for quality improvement (van Leersum et al., 2013;Ivers et al., 2012). Therefore, since the improvement of patient safety and quality of care are both high on the international political agenda, the registration of clinical data is indispensable in the current duties of a clinician (Dreyer & Garner, 2009). Besides, this data collection is also needed to support the possibility to conduct clinical research studies. As incomplete or incorrect data is not usable for the assessment of quality or for conducting clinical research, it is essential to engage and motivate clinicians to register.
In this context, gamification elements may offer opportunities to motivate and engage doctors to participate in medical registries. Gamification can be explained as the use of game elements and techniques in existing applications or in nongame contexts, to motivate and engage users with a system (Morris, Croker, Zimmerman, Gill, & Romig, 2013). Gamification focuses on making necessary and annoying tasks more enjoyable through a positive approach (Dithmer et al., 2015). During the last few years gamification is used within a broad variety of domains, such as finance, health, education, news and entertainment, for example by the earning of badges, points and achievements when completing specific tasks (Deterding, Dixon, Khaled, & Nacke, 2011). In a similar vein, serious gaming is increasingly used to train doctors technical and non-technical skills relevant to the surgical field (Graafland, Schraagen, & Schijven, 2012). To note, gamification and serious gaming are two different concepts, but show many similarities. Serious gaming refers to games for non-entertainment purposes whereas gamification refers to the use of elements from games in non-game contexts (Deterding et al., 2011).
Despite the increased popularity of gamification, to the best of our knowledge, this strategy had not been used yet in the context of medical research to motivate an engage physicians participating in medical studies, and even more interesting, on the impact on clinical outcomes. However, recent studies showed that the use of gamification impacted residents' engagement in simulation training, and motivated heart patients as a part of a rehabilitation program (Kerfoot & Kissane, 2014;Dithmer et al., 2015). Furthermore, it is noted that the impact of gamification in health-related contexts has achieved significant results (Pereira, Duarte, Rebelo, & Noriega, 2014). Therefore, the introduction of gamification elements could potentially be helpful to motivate clinicians to register their data completely and correctly. In this light, it is well known that providing audit and feedback to the clinician leads to improvements in professional practice. Audit and feedback is defined as a summary of clinical performance over a specified period of time preferably leading to clinical actions (Ivers et al., 2012). However, there is lack of knowledge about what kind of feedback is most effective to increase insight in personal performance and to increase engagement to register medical data (Ivers et al., 2012;Maruthappu et al., 2015). The high workload of surgeons is a main reason for non-participation in medical registries and aspects such as lack of support and feedback, but also lack of rewards and recognition are cited as reasons to not participate in medical registries (Albers & Sedler, 2004;Rahman et al., 2011). Hypothetically, the addition of gamification elements in the provision of feedback could enhance the effect of feedback because of the positive enforcement of gamification.
The aim of this study is to determine the effect of additional gamification elements in a web-based registry system for laparoscopic hysterectomy (LH) in terms of engagement and involvement of gynecologists to register their outcome data and to determine if gamification elements have any effect on clinical outcomes.

Design and Participants
The CONsolidated Standards of Reporting Trials (CONSORT) statement was followed to describe the design of the study (Schulz, Altman, & Moher, 2010). All Dutch gynecologists who perform laparoscopic hysterectomy (LH) were eligible for participation and were asked to register all their consecutive LHs between April 2014 and November 2015 at a newly introduced web-based application, https://www.QUSUM.org (QUality indicator of surgical performance in minimally invasive surgery). Figure 1. The individual, direct feedback graphs as provided to participants from both groups in the study Explanation of graphs: If a surgeon performed better than expected, the line drops. If a surgeon performed worse than expected, the line rises. The y-axis shows the cumulative difference of the surgical performance compared to the benchmark.

Intervention Group
The intervention group also received access to the web-based application. Besides the individual, immediate feedback graphs as well gamification elements were shown (   A. Activity tracker, showing the latest contributions to the QUSUM study by the participants in the intervention group. Here, participants are able to see that other participants (from the intervention group) are contributing to the total score.
B. After register a new procedure, a message pops-up that shows how many points the participant had earned by registering this very procedure.
C. The user is part of the QUSUM collective (represents the scores of all registered procedures of all participants). When the user performs better than expected, a contribution is made to the collective scores.
D. Overview of personal achievements.
E. Monthly comparison of QUSUM Collective Score; when the QUSUM collective (i.e., all participants in the intervention group) performs better than the national benchmark this is indicated by a green scale, when the collective performs worse this is indicated by a red scale.
The gamification elements were developed by experts from the Institute of Psychology, Leiden University, and were selected to induce competition, motivation and collaboration based on positive enforcement. They consisted of three key components; 1. points that could be earned when there was registered procedure had less blood loss and/or less operative time than what could be expected based on the case mix, and points for the contribution of knowledge to the study (i.e., by providing additional information concerning the procedure, Figures 3A, 3B and 3C), 2. Participants could earn individual achievement badges for their general contribution by registering procedures ( Figure 3D) and, 3. Insight in a monthly collective score which represented the aggregate scores of all registered procedures per month of all participants in the intervention group. Each month the scores of the QUSUM collective were compared with existing national benchmark data ( Figures 3C and 3E).
In addition to these key components, an activity tracker showing the latest contributions of all participants was visible at the homepage ( Figure 3A). Also, after entering a procedure, a message popped-up showing how many points the participant had earned ( Figure 3B).
The application complied with NEN 7510 standards (Dutch certification regarding informatics and security in the healthcare field) and was approved by the privacy officer at Leiden University Medical Center. Since no identifiable patient data was requested, this study was exempted from approval by our Institutional Review Board at Leiden University Medical Center.

Outcome Measure
The primary outcomes of this study were engagement and involvement of participants to register their procedures, which were assessed by the use of a web-based survey. This survey was developed by the Institute of Psychology at Leiden University and was send to all participants at the end of the study period (November 2015). A Likert scale of 1 to 5 was used (never to always, never to a great deal, not at all too very, not at all to always). It is shown that different types of motivation can be most objectively answered using Likert-scales (Ryan & Connell, 1989). Furthermore, involvement and engagement were assessed by the behavior of users as logged by the application (e.g., number of login sessions, number of active views of features of the application). Furthermore, during initial registration, the users were asked to rate their motivation to participate in this study on a Likert scale 1 to 5 (e.g., very low to very high), to enter the number of LH's performed yearly (their annual surgical volume), to enter the total amount of LH performed during their career (their experience) and to enter the number of years they were performing LH's ( Table 1).
As secondary outcome was selected; the effect of the gamification elements on the surgical performance, which was defined as operative time, intraoperative blood loss, and complications. Operative time was described as the number of minutes between the first incision and insertion of the final stitch, blood loss was measured in millilitres, and complications were registered as determined by the Dutch Society of Obstetricians and Gynecologists (Twijnstra, Zeeman, & Jansen, 2010

Sample Size
A prospective sample size calculation was not applicable for this study, since the intent was to include as many gynecologists as possible. We consider a retrospective sample size calculation as arbitrary.

Randomization
During initial registration participants were randomly assigned to either the control or the intervention group using computer-generated randomization. Block center randomization was applied, meaning that gynecologists from the same center were allocated in the same group, in order to avoid notification of the other study condition when discussing results with direct colleagues. Participants were included for analysis when at least one procedure was entered in the application.

Statistical Methods
For the statistical analysis, SPSS version 22 (IBM Corp., Armonk, NY) was used. Mean values of surgical outcomes were calculated with their standard deviation (SD). Differences were statistically significant at p<0.05.
To account for clustering of data from multiple entered procedures by a single surgeon, generalized estimation equations were used for the analyses of differences of surgical outcome between the two groups. Logistic regression was used to analyse the difference between the groups with respect to their activity on the application. Dependent variables used in this model included total number of entered procedures and study motivation of users. The questionnaire regarding engagement and involvement consisted of multiple subscales, measured on a 5-point Likert scale. For each subscale the Kaiser-Meyer-Olkin (KMO) measure was used to assess the general factor structure. Then, for each subscale that met the KMO criterion of>0.5, a factor analysis was performed to assess which items to include in the subscale. Items with factor load<.30 were removed. Finally, the reliability of each subscale was calculated using Cronbach's Alpha. We adopted the threshold value of .70 or above to consider the subscale as reliable. In order to assess the influence of condition on the various subscales, a MANOVA was conducted.

Results
From April 2014 to November 2015, a total of 71 Dutch gynecologists enrolled in the study and entered one or more LH procedures. Of the participants, 37 gynecologists were randomized in the gamification group and 34 participants in the control group. A total of 53 participants (75%) completed the survey, of which 28 of the gamification group and 25 of the control group (Figure 4).

Figure 4. Flowdiagram of participants
A total of 1833 LHs were registered. The mean ±SD number of entered procedures of participants was 27.5±23.1 (Table 1). Surgical volume and experience of both groups is shown in Table 1.

Engagement
The observed mean (±SD) for the engagement subscale was 2.44±0.96 for all participants combined (Table 2). No significant difference was found between the two groups; 2.34±0.87 for the intervention group versus 2.56±1.05 for the control group (p=0.41).

Involvement
For the involvement subscale a score of 3.49±0.83 was observed for all participants combined (Table 2). No significant difference was found between the two groups, respectively 3.63±0.57 versus 3.33±1.03 for the intervention and the control group (p=0.19). The majority of both groups considered the contribution to the QUSUM study as (very) important (Likert scale of 3.96±1.0).

Activity on Application
No significant difference for any activity on the application was observed between the two groups (Table 3). A mean of 22.2±18.8 login sessions was observed for all participants combined.

Surgical Outcomes
A significant difference was observed for mean (±SD) operative time. The intervention group showed longer operative time (108±42 minutes) than the control group (101±34 minutes) (p=0.039). For blood loss and complications no significant difference was observed between the two groups, 122±164 vs. 144±173 mL and 4.8 vs. 8.7%, for respectively the intervention and control group (Table 4).

Ease of Use of Application
A significant difference was observed regarding the clearness of the possibilities of the QUSUM application. Participants in the intervention group, who used the application with gamification elements showed lower scores (3.65±1.2) than the control group (4.24±0.7) (p=0.019) (Table 5). Overall, registering procedures in the applications is considered for the majority of users as (very) easy (4.46±0.7). The individual, direct feedback graphs for surgical outcomes (Figure 1), which are provided in both groups, are considered clear (overall score of 3.56±1.2) and useful (overall score 3.79±1.1).

Conclusions
The addition of gamification elements in a registry system did not enhance the engagement and involvement of clinicians to register their clinical data. In addition, our results showed that the features of the application were significantly less clear for the users in the gamification group, which can be explained by the fact that this version of the application consisted of many more elements that need to be understood. This may suggest that easiness and simplicity of an application is more important to engage users. Furthermore, if we look at surgical outcomes, we observed a significant difference in operative time in favor of the group without gamification elements. Therefore, our results demonstrated that the addition of gamification elements did not show any advantages and may even imply that the gamification elements could distract users from the primary goal of the application, which is the provision of direct feedback to the surgeon.
In general, the application introduced in this study was rated as very useful and (very) easy to use by the majority of participants (Table 4). Therefore, we recommend that registries should be simple and exclusively collect data that is truly relevant and usable. In addition, we assume that gamification elements will also be distracting in more comprehensive registry systems.
Another important result is that the majority of users believe it is important to contribute to a study and consider that the registration of procedures in the application has positive impact on their clinical performances (Table 2). This suggests that the participating clinicians are already aware about the necessity of registering clinical data, and therefore, the focus should be placed on making this easier and less comprehensive for them. Furthermore, to reduce the extensive administrative workload of clinicians, a future development of new registries should be the possibility to implement these in existing data systems.
Strength of our study is the fact that this is, in our knowledge, the first study that determined the effect of gamification elements in a randomized control design study. Considering the current extensive administrative workload for clinicians this is an interesting topic (Woolhandler & Himmelstein, 2014) and attempts to make registries more fun and more easy for clinicians are increasingly relevant. Although we conducted our study with the use of a registry application for laparoscopic hysterectomy, we consider our results to be generalizable to registries of other kinds of procedures and specialties.
The benefits of additional gamification elements are previously demonstrated in other domains of healthcare (e.g., simulation training for residents and patient engagement to rehabilitation) (Kerfoot & Kissane, 2014;Dithmer et al., 2015). However, unexpectedly, our study did not show any benefits from the applied gamification elements in the domain of doctor's engagement and involvement to register clinical data. In comparison to other fields, the use of gamification elements in medical (research) registries has several limitations regarding the choice of elements. As privacy issues limit the boundaries of possibilities in gamification, we also must be careful with the use of elements, which are considered to be fun respecting patient related outcomes. Furthermore, in many cases the financial resources are limited to properly design a registry system and their gamification elements.
A possible limitation of our study could be the fact that the used gamification elements were not intuitive enough for users to understand, which could result in the opposite effect of their initial goal. A potential solution for this problem is to instruct all users first about the exact meaning of the gamification elements. However, this will shift the accent to games instead of registration. And in daily practice this is probably a mission impossible, since we will all recognize the fact that reading of a detailed guideline and/or instruction is cumbersome and annoying for clinicians. Therefore, we consider that gamification elements can only have a chance of success, when the meaning is completely intuitive and no detailed explanation required. Another potential weakness is the relatively low number of participants, however since the participants entered more than 1800 procedures they were sufficiently exposed to the gamification elements.
Much research has demonstrated the positive impact of feedback on clinical outcomes and professional practice (Ivers et al., 2012;Maruthappu et al., 2015;Foy et al., 2005;. Therefore, another possible explanation of our results might be the following: the individual, immediate feedback graphs on surgical outcomes, which were shown in both groups (Figure 1), may already have provided enough positive enforcement to involve users. As a consequence, no difference between the groups was observed.
With this study we investigated the question, how to make a registry system more attractive for clinicians to register data. To conclude, our study showed that the addition of gamification elements in a registry did not affect engagement and involvement of clinicians. Based on our results, we advise that registry systems for clinical data should be as simple as possible with the focus on the main goal of the registry. This is especially true considering the increased pressure to register a large amount of (clinical) data currently; irrelevant features, which can distract users from the primary task, should therefore be minimized.

Funding
This study received no funding.