Wikipedia Editing Courses at Three US Medical Schools in the 2017-2018 Academic Year

Introduction: Most medical students use Wikipedia, yet most medical schools do not train students to improve Wikipedia or use it critically. Methods: During academic year 2017 2018, the University of California San Francisco School of Medicine, Medical University of South Carolina & University of Central Florida College of Medicine offered credit-bearing Wikipedia-editing courses to 4th year medical students. We sought to assess student experiences across these schools and courses. Results: 78 students participated in the courses, making 3,368 edits, adding 155,100 words to 71 health-related Wikipedia pages. Collectively student contributions were viewed 2,688,500 times (37,866/page) during active dates of each course. Feedback from the 18 students who completed our study survey (23% response rate) reflected that students 1) increased their perception on Wikipedia reliability, 2) increased their ability to practice evidence based medicine, 3) were positively inclined to encourage peers to also edit Wikipedia, and 4) desired to contribute to public good. All students felt they improved their respective articles, and all but one felt that editing Wikipedia was a good investment in the future of healthcare. Despite our low response rate, qualitative comments suggest several students felt their courses impacted their professional identity as future physicians. Conclusions: We encourage the implementation of similar Wikipedia-editing courses across all health professional schools. Study replication will help clarify generalizability of these findings. Standardizing outcome metrics (e.g. impact on students, impact on Wikipedia quality, and impact on readers of Wikipedia) will allow more robust Joshi M, Verduzco R, Yogi S, Garcia M, Saxena S, Tackett S, Dexter N, Dempsey A, Whitaker E, Azzam A MedEdPublish https://doi.org/10.15694/mep.2019.000146.1 Page | 2 assessment of the impact of Wikipedia-editing courses and assignments across health professional schools globally.


Introduction
The public commonly uses Wikipedia for general information. In addition, 50-70% of physicians and over 90% of medical students use Wikipedia (Heilman and West, 2015). As of March 2017, Wikipedia had nearly 30,000 articles on medical topics in English, and 164,000 in other language Wikipedias . Collectively these articles are viewed more than 10 million times per day (Shafee, Mietchen and Su, 2017). However, the website carries a negative stigma in content integrity and is criticized for having a lack of traditional editorial controls (Metcalfe and Powell, 2011). To protect content from inaccuracies (e.g. vandalism, self-promotion, factually incorrect content), Wikipedia's quality control processes include the use of watchlists, recent change tracking, bots, special content page protection, filters, blocking, and banning problematic editors (Metcalfe and Powell, 2011). Above all, it is fundamentally the work of human volunteer editors that has kept Wikipedia's content reliable to date.
Extensive collaboration is necessary for a Wikipedia article to become accurate and comprehensive. Wikipedians aggregate around WikiProjects--defined as collections of individuals interested in particular topics on Wikipedia. WikiProject Medicine is a group of about 300 active experienced editors, many of whom are health professionals . Individual volunteers within WikiProject Medicine work to improve the currently 43,333 English Wikipedia health content pages through collaborations with libraries, scholarly journals, and organizations such as Cochrane, and Cancer Research UK (Wikiproject Medicine, 2018a). Translators Without Borders translates these health-related English language Wikipedia pages to other language Wikipedias (Wikiproject Medicine, 2019a). "Internet-in-a-box"-a low-cost digital library designed to provide access to the internet offline-works with WikiProject Medicine to provide access to Wikipedia's health-related content in places where access to the internet is limited or controlled (Wikimedia, 2019). Lastly, WikiProject Medicine has designed an offline Medical Wikipedia app available in 10 languages (Wikiproject Medicine, 2019b). When aligned with openaccess publishing and indexing, this "collaboration chain" provides communities around the world access to high quality health information in their preferred languages and for free (Heilman et al., 2011). Consequently, Wikipedia has the capacity to be more than a resource for general information--it is a "public health platform" that makes it possible for established and health-professionals-in-training to contribute to population-level health (Heilman et al., 2011).The current rate-limiting step is the limited number of active volunteer editors participating in WikiProject Medicine.
The Wiki Education Foundation (Wiki Ed) is a non-profit organization promoting Wikipedia integration into higher education coursework (Wiki Education Foundation, 2018a). Wiki Ed has a dashboard through which instructors can designate and customize learning modules for students, track student progress and organize other course content (such as work-in-progress sessions and assignment deadlines) (Wiki Education Foundation, 2018b). Since 2010, over 63,000 students across over 500 universities in North America have contributed over 30 million words to over 68,000 Wikipedia articles (Blumenthal, 2017;Wiki Education Foundation, 2018b). Editing Wikipedia within higher education provides opportunities to improve open access health content used for professional and personal reasons alike. Health professional students who edit Wikipedia apply their newly acquired knowledge and information-seeking and appraisal skills. And their effort goes onto a platform that is public-facing, which challenges these emerging professionals to present technical content in comprehensible terms. Furthermore, faculty and librarians can customize Wikipedia-editing courses and/or assignments to local contexts. Joshi M, Verduzco R, Yogi S, Garcia M, Saxena S, Tackett S, Dexter N We published an earlier innovation report summarizing early experiences of the first US medical school course dedicated entirely to Wikipedia-editing (Azzam et al., 2017). In writing for Wikipedia, those medical students found it challenging to balance the needs of the general population with health professional audiences. Those results were limited, as it was a single institution study with early adopting faculty and medical students. But by November 2018, at least 10 health professional schools across 5 countries and 6 health professions had developed Wikipedia editing courses or assignments, most using the Wiki Ed course dashboard. We sought to evaluate the impact of this emerging curricular approach by querying students who completed some of these courses.

Course Structures
The Wiki Education Foundation restricts its supportive function to universities in North America. Consequently, for our study we limited inclusion criteria to US medical schools that offered credit-bearing courses for fourth year medical students dedicated primarily or exclusively to editing health content on Wikipedia. In the 2017 -2018 academic year, the entire eligible pool included courses at University of California San Francisco (UCSF), Medical University of South Carolina (MUSC) and University of Central Florida (UCF). The MUSC elective was offered once, the UCSF elective twice, and the UCF elective offered 3 times. These courses were all co-led by faculty and librarians at each medical school. Each month-long course began with an in-person orientation and each instructor assigned Wiki Education Foundation-produced learning modules (e.g. training on plagiarism, WikiProject Medicine style guidelines, how to add citations, finding open-access images). Each course additionally held synchronous workin-progress sessions weekly (though students were allowed to participate remotely via video conference). During these work-in-progress sessions, faculty and medical librarians provided task guidance and answered general questions (e.g. formatting, what content to include and where, what content to deliberately exclude, how to engage with Wikipedians, etc.).
Each course was designed and executed via the Wiki Education Foundation course dashboard. A screenshot of one of these courses' dashboard is provided as Figure 1.  In some courses, students could only select articles from a restricted list provided by their faculty instructors. In all courses, students were encouraged (but not required) to select high-importance but lower quality articles--as cataloged by WikiProject Medicine (WikiProject Medicine, 2018a). Students were also permitted to work in teams. Through the Wiki Education Foundation course dashboards, faculty tracked the progress of individual student editors as well as efforts of the course en masse regarding aggregate effort. All courses required students to peer review at least one classmate's work. The final day of each of the courses required students to present their revised article to their classmates and reflect on their accomplishments and lessons learned. Detailed course structures are publicly available through Wikipedia itself (Wiki Education Foundation, 2017a;Wiki Education Foundation, 2017b;Wiki Education Foundation, 2017c;Wiki Education Foundation, 2018c;Wiki Education Foundation, 2018d;Wiki Education Foundation, 2018e).

Methods
The UCSF Institutional Review Board approved this data collection protocol across the 3 participating medical schools. While each school conducted its own internal course evaluations, we sought to assess variation and similarities across all courses. Post-course student feedback was therefore evaluated via a Qualtrics survey designed by the authors in partnership with Hanover Research. Hanover Research is a research and analytics firm that partners with higher education institutions to provide custom analysis for academic solutions (Hanover Research, 2018). Our research was conducted with the explicit permission of the local medical school faculty. But because our eligible student pool was quite small, we chose to use Hanover Research to assist with data collection to maximize student candor and minimize any perception of obligation to participate in our study. The 16-item mixed-methods survey included quantitative and qualitative inquiries through Likert-style and free-text items respectively. We did not pilot test or validate the survey items, which are provided as Appendix 1. The survey was launched on May 8th, 2018 and results were collected electronically. Non-responding students received intermittent email reminders to complete the survey. Data collection was completed by June 8th, 2018. *Words added ranges approximated through bytes added, with a conversion ratio in English of 5.175 bytes per word. This ratio is used across all Wiki Education Foundation courses.

Survey Results
Eighteen students completed our survey (response rate = 23%), including students from all 3 schools (3 from MUSC, 7 from UCF, and 8 from UCSF). All eighteen students felt confident they improved the content of their Wikipedia article and all would recommend their course to other students. All but one agreed that improving Wikipedia is a good investment in the future of healthcare. The cohort believed participation in their courses led to an increase in likelihood of editing Wikipedia in the future (n=15, 83%), ability to practice Evidence-Based Medicine (n=15, 83%), capacity to write in clear terms (n=15, 83%), perception of Wikipedia quality (n=14, 78%), and likelihood of recommending Wikipedia as a learning resource (n=14, 78%). Students ranked their reasons for selecting their respective articles. Figure 3 summarizes key highlights of the survey results.  When asked to describe the reliability of Wikipedia for patients, many students felt Wikipedia was only moderately to slightly reliable. One student wrote "It's moderately reliable for health information, but I would only use it as a starting point to talk with your doctor rather than using it to guide decisions." Reasons for future editing varied, but the most recurring motivators were 1) to help the general public and 2) if the Wikipedia article was related to their field of interest. The most valuable lessons students learned were 1) the use of Wikipedia for quality/relevant information and 2) how the Wikipedia editing community operates. Five of the eighteen felt the class was so impactful that it influenced their professional identity as future physicians. Lastly, the two most frequently recommended improvements were 1) a more structured learning experience and 2) more peer/instructor communication. Exemplar quotes from students' qualitative responses are provided in Table 2. What do you feel is the most valuable thing you learned from this course?
• "I appreciated learning about the quality ratings on Wikipedia articles. Since this course, I have reviewed the quality ratings of articles I have used for medical purposes before referencing them to get a sense of the quality of the information provided." • "Learning the inner workings of Wikipedia and the rigor with which the community edits the available content. The course provided much clarity into both the strengths and weaknesses of Wikipedia, and helped me reframe how I perceived Wikipedia as a distributor of medical content." • "Understanding the concept of health knowledge accessibility as a major issue in the realm of medical care --practicing writing concisely and in lay-language to make medical knowledge more digestible." • "Delving deep into a focused topic as well as learning how to synthesize medical topics into a way understandable to the general public." What would have improved your learning experiences throughout this course?
• "More personal connection to medical students and/or healthcare providers who rely on this resource. Do they have any particular requests for articles that they would want updated or improved?" • "A factor that could not be controlled, but [hearing from] previous participants in the course…could have helped improve the advice given by instructors." • "Actually editing content and spending time engaging with the Wikipedia community was the most valuable aspect of the course. The course did an excellent job exposing us to both, but anything that can increase master Wikipedian exposure would improve the experience further. • "Knowing that patients frequently read articles to learn more about their own health and to contribute to the efforts of Wikipedia to provide access to educational materials to underserved communities and third world countries." • "Some form of continuing education credits or recognition; professional societies with improving Wikipedia as an objective; CV publication" In the language you would use speaking to your patients, describe the reliability of Wikipedia's health-related content.
• "The articles on Wikipedia do contain a good deal of accurate information. This information comes from reliable sources which doctors use to remain up to date on information. Those sources are the best place to look for information, and Wikipedia is a great jumping off point. You are always invited to consult with me as your physician to explain anything you don't understand or want to verify." • "When it comes to common, well published, medical conditions/topics I would trust what you read on Wikipedia • "I believe Wikipedia is an excellent resource to quickly find information from well-organized, succinct articles and that those in the medical profession can greatly contribute to public health by dedicating time to editing articles." • "There is more that goes into editing than I thought." • "Honestly, my opinions about the quality of Wikipedia probably decreased through this course. After seeing some of the poorer quality articles on Wikipedia, it just made me more aware of the fact that you can't believe everything you read on Wikipedia." Please describe how this course influenced your professional identity • "I hadn't really thought about a responsibility to use my training, knowledge, expertise, etc. to cultivate a media environment with more accurate, actionable information for the general population." • "I feel that contributing to the availability and reliability of medical information is a part of my role as a physician."

Impact on Medical Students
Based on both quantitative and qualitative responses, these Wikipedia editing courses appear to have demonstrated substantive impact on medical students, as well as their perceptions of the quality of Wikipedia as an open-access resource for health information. Students are challenged to translate their technical medical knowledge into language that is comprehensible, succinct and thorough. By interfacing with Wikipedia, students expand their physician identities in the context of a developing, digital workspace, as well as a collaborative platform involving health professionals and other editors. Participation in these courses may be the beginning of extensive careers involving improvement of Wikipedia's health content.

Impact on Wikipedia
We did not systematically assess the impact of students' work on Wikipedia quality. However, based on personal communication with the course faculty of record at each institution, during the time students were actively editing, their selected pages appeared to be significantly improved. WikiProject Medicine's community of volunteer editors have consistently agreed with the edits and additions being made by students enrolled in these courses. Acknowledging the relatively recent course end dates, the contributions made by students do not appear to be actively vandalized or deleted.

Impact on Medical Education
These courses are part of several larger movements in higher education. Open Educational Resources (OER) are defined as free and openly licensed educational materials that can be used for teaching, learning, research, and other purposes (Hewlett Foundation, 2018;Creative Commons, 2018). A relatively newer term is Open Educational Practices (OEP). One definition of OEP is the use of OER for teaching and learning in order to innovate the learning process (Wikipedia, 2019). Lastly, Free Open Access Medical Education (FOAM) can be defined as a community of open access resources for learning in medicine (Scott et al., 2014). All of these movements are influenced by larger forces such as crowdsourcing, social media, and the penetration of technologies in developed (and increasingly developing) countries. As these movements grow, there are calls for consortia of medical schools to pool resources (Le and Prober, 2018) as well as examples of networked communities of health professional students using free videos and adaptive learning platforms to make learning more efficient (Menon et al., 2017;Tackett et al., 2018). We believe these are all positive developments in the longitudinal trajectory of medical education.

Limitations
There are several limitations to our work. These results represent only three US medical schools, and we had a very low response rate to our survey. This may have been due to the unexpected source (Hanover Research), but also due to the timing in students' academic calendars (all were imminently graduating from medical school). Despite having student responses across all three participating schools, the low response rate may not represent an accurate distribution of opinions of the students on their respective courses. Furthermore, variations in these responses may arise from the inconsistencies between courses across different instructors. Each instructor likely implemented different teaching methods, causing even more variation between individual student experiences. We also did not independently review the content of students' work for accuracy (though we presume all local faculty did). We did not grade the Wikipedia articles for quality, either before or after the students' contributions. Despite these limitations, we believe that the qualitative data provides ample and important insight into student perspectives on contributing to Wikipedia within formal medical school curricula across several medical schools.

Areas for future study
Through the creation of the Wiki Education Foundation dashboard, Wikipedia editing classes and coursework and have become more accessible to students and faculty alike. Resources such as instructional videos and interactive learning modules in the Wiki Education dashboard, and collaborative student spaces within the Wikipedia editing interface can appeal to Millennial learners who place a premium on flexibility and active educational experiences (Eckleberry-Hunt, Lick and Hunt, 2018). These types of collaborations create a model environment for students across various health-related fields to work together to improve Wikipedia's large, free information library without geographic barriers.
Wikipedia editing as a teaching and learning strategy has expanded into other Health Professional schools, as has centralized tracking of progress. The Wiki Education Foundation has created an aggregate campaign called "Students in Health Professions." At the time of this writing (May 2019), the campaign included 857 students across 27 courses. Collectively those students added 613,000 words and 84 images to 406 health-related Wikipedia pages that had been viewed 16.1 million times since students began contributing (Blumenthal, 2017). Future multiinstitutional and even global collaborative studies might systematically evaluate these courses' impacts on their students, their faculty, and on the quality of Wikipedia itself.
Currently, the Wikipedia editing courses use the dashboard feature to track individual student contributions and assess metrics such as number of edits, words, images and citations added. In order to evaluate each article, WikiProjects have previously created an assessment rubric of article quality. The grading scale ranges from "stub" to "featured article" and includes explicit criteria for the promotion of an article from one grade level to the next.
Despite the presence of this standardized Wikipedia article quality rubric, students' work in each medical school course is graded based on subjective criteria by individual instructors. In most cases the courses were pass-fail (personal communication with course instructors). The Wiki Ed course dashboard provides a suggested mechanism to grade student's efforts on Wikipedia assignments, but none of the faculty of these courses elected to use that grading mechanism (personal communication).
Standardizing both the assessment of students and their Wikipedia-editing efforts would provide more accurate assessments of student capacity to meaningfully contribute to Wikipedia's health related content. Yet standardization would also carry some risk, as the amount of effort necessary to improve articles varies based on multiple factors, such as the breadth of the topic, the baseline completeness prior to course start, and the student's year in his/her training program. This again represents a rich area for potential future studies. In the 2017 -18 academic year, three US medical schools offered credit-bearing courses focused primarily on Wikipedia-editing. Our survey of students participating in those courses suggests that these efforts had substantive impacts on students and Wikipedia quality. We invite other health professional schools to join this movement, as we believe these efforts advance Wikipedia's mission--to act as a comprehensive written compendium that contains information on all branches of knowledge (Wikipedia, 2018).

Take Home Messages
Wikipedia is a relied upon and increasingly reliable resource for medical information There is a growth of infrastructure to execute and evaluate Wikipedia-editing courses and course assignments Several medical schools have incorporated Wikipedia-editing as a formal part of their curricular offerings Medical students editing health content on Wikipedia is a learning and collaborative practice Wikipedia-editing courses within formal medical school curricula broaden students' professional identities as future physicians

Notes On Contributors
Mihir Joshi, Rafael Verduzco, Sara Yogi, Maite Garcia, and Sanskriti Saxena were all undergraduate research assistants with Dr. Azzam at the University of California, Berkeley at the time of their participation in this research.

Sean Tackett is a General Internist at Johns Hopkins Bayview Medical Center and Research Director for
Osmosis. His career aspiration is to contribute to improvements in the quality of health professions education internationally. Sean currently teaches curriculum development to Johns Hopkins faculty, has authored online modules on curriculum development and learning theories for Johns Hopkins School of Medicine, and has taught educational skills to faculty from China, Oman, Saudi Arabia, Vietnam, Syria, and Brazil. He also practices hospital medicine and teaches medical students and residents on the medical wards.
Nadine Dexter is the Director of the Health Sciences Library and Director of Medical Informatics at the University of Central Florida College of Medicine (UCF). She also is a core faculty in the Wikipedia-editing courses at UCF.