The Promotion of Hearing Health through Wikipedia Campaigns: Article Quality and Reach Assessment

This case study examined the feasibility, reach, and potential impact of using Wikipedia as a tool for hearing health promotion. Activities involved editing existing Portuguese-language Wikipedia hearing health articles, as well as translating English-language hearing health articles to Portuguese during the Wiki4WorldHearingDay2019 and Wiki4YearOfSound2020 online campaigns. The Wikipedia efforts that took place in Brazil were carried out by 10 volunteer undergraduate students in Speech-Language Pathology and Audiology at the Federal University of Santa Catarina, in Brazil. Among new and existing Wikipedia articles, the group edited 37 articles, which attracted more than 220,000 views during the set tracking period. Students were responsible for 60% of the Portuguese-language edits during the Wiki4WorldHearingDay2019 campaign and more than 90% of the Portuguese-language edits during the first half of the Wiki4YearOfSound2020 campaign. Moreover, the quality indexes for pages either created or edited were improved in all situations by registering an increase rate ranging from 33% to 100%. Wikipedia-centered activities expanded the availability of quality scientific content, written in plain language, to the public. Students worked together in order to select topics, assess existing information, validate it, create new content, and share information—steps that contributed to the mission of health promotion and knowledge dissemination for the benefit of society.


Introduction
The lack of quality information about health conditions and existent health services is considered an obstacle to promoting public health. This challenge is demonstrated with hearing health [1], considering the several year gap between the first signs of hearing difficulties and the subsequent search for healthcare [2][3][4][5][6]. New technologies, expanded access to telehealth, and open communication are needed to change this scenario [7]. Information available in simple and accessible language can help people understand and implement health information into their daily lives. Access to free information is recognized by the Centers for Disease Control and Prevention as a necessary step in promoting health in general [8].
choice of Wikipedia account names. They were informed of the public and transparent nature of the assignment and the plans to analyze the experience in a feasibility/case study.
A formal training process was developed and implemented. As participation was voluntary and did not have any formal grading or other performance evaluation, the main indicator of feasibility was retention. Attention, participation, and effort were also indicators of the viability of the approach. The project was publicized among the students through virtual channels, and interested parties signed up. Participants were eligible to participate if they could dedicate a minimum of two hours a week, which included participating in project meetings.
Ten undergraduate students volunteered to participate in the first proposed activity, which was part of the Wiki4WorldHearingDay2019 campaign. They were invited to contribute to a second campaign, Wiki4YearOfSound2020, and eight accepted. These two campaigns were selected because of the prominence of the organizing institutions and their common objectives to expand outreach and increase awareness among the widest possible audience. We noted that the campaigns also give students not only the motivation but also the opportunity to exercise new technical skills to expand the scientific content in hearing health while being able to measure the impact of their activities with objective metrics.
Study activities involving Wikipedia began in January 2019 and included the following: (1) locating and evaluating specific content on the intersection of audiology and environmental and occupational health; (2) discussing scientific evidence in these topic areas; (3) engaging in the health education process; and (4) introducing students to a publicdirected scientific writing experience. In a partnership with researchers from NIOSH, the agency overseeing the two Wikipedia global online campaigns, the group focused on topics related to the promotion of hearing health.
Students took an 8 h training course offered by experienced Wikipedia writers and editors, called Wikipedians, from the Federal University of Santa Catarina. They used educational materials and tutorials developed by the Wiki Movement Brazil User Group (WMB) [24], the Wikipedia Education Program [25], and other collaborators. Participants learned about tools that are available for translating from Wikipedia: Translation [26] and from the training library of Wiki Education. This Wikipedia training covered the theoretical framework of the platform while hands-on experience demonstrated the processes for editing, translating, and developing content [27]. In addition, the Wikipedia Crowdsourcing Group from the University of Sao Paulo held online workshops and technical support training sessions throughout the effort. The main objective of the training was to show the students how to locate and evaluate sources for their verifiability, to discover associated content gaps in Wikipedia gaps, and to improve the content on the platform.
After the training was completed, students individually carried out writing assignments, which were tracked as part of the Wiki4WorldHearingDay2019 campaign. The campaign platform (available in 17 languages) had guidance on how to identify topics and improve or translate existing Wikipedia articles. This campaign was cohosted by the World Health Organization (WHO) and several professional organizations. A project coordinator provided online weekly guidance to students.

Analysis Methods for Reach and Quality of Student Contributions
The number of pageviews before these activities began, as well as changes in pageview trends afterward, was tracked by a set of online tools developed, maintained, and hosted by the Wikimedia Cloud Services team and volunteers [28]. The tracking tools used were Outreach Dashboard, Pageviews Analysis [29], and xTools [30]. The tools, which were hosted at Wikimedia Toolforge Services, performed analytics, administered bots, and ran specific web services, as well as created other tools [31]. The "number of views" metric provided information on the interest from Wikipedia readers on a specific subtopic [11,32].
The Outreach Dashboard was used to track students' activities and determine the amount of content both in kilobytes (kB) and in number of words added to each page during the campaigns. The dashboard also provided the number of articles created and edited, along with the number of citations and other media files added. The Pageviews Analysis tool and xTools were both applied to the precise number of pageviews in a given period: 30 days before activities began and 30 days after they ended.
Beyond the quantitative analysis of pageviews, an evaluation criterion for both improved and translated contents was provided. An online tool that uses semantic data tracked the evolution of content through article quality assessments based on Objective Revision Evaluation Service (ORES) technology [33], developed and maintained by the Wikimedia Machine Learning team (shown in Figure 1). Essentially, it is an artificial intelligence service that runs on the backend and aims to provide machine learning features for an ecosystem of tools available on Wikimedia projects [34]. These tools make it possible to automate critical and time-consuming work in the context of Wikimedia projects [35]. Their most common applications include counter-vandalism tasks and quality predictions of new edits and articles. For the latter, ORES adopts a combination of open data and open-source machine learning algorithms to train models and create scores, which may vary depending on the Wikipedia project and its respective quality grading scheme. ORES scores help predict the quality of edits as they are made to the pages, as well as the quality of an article based on each given history revision (RevID).  Thus, the articlequality model of the ORES UI tool was applied in two circumstanc on each page: 1. To compare the index of quality after the last history revision prior to student activ ties to the last history revision after the student interventions to find the improv quality. 2. To obtain the index of quality after the last history revision, conducted right after t end of student activities (for the created/translated content).
In the case of translated pages, only the revision history was used to obtain the qu ity index. Then comparisons regarding the evolution of the indexes between the two form of activities (content creation/translation vs. content improvement) were carried out determine the effectiveness of each approach and its impact.

Global Online Campaigns
Activities from the Wiki4WorldHearingDay2019 campaign were tracked from 21 Ja uary 2019 through 31 March 2019. Overall, the results indicated that 74,000 words we contributed to 90 existing and 7 new Wikipedia articles, and 21 images were donated the open access repository WikiCommons. These expanded (edited) and new articles r ceived more than 2.8 million views during the 2-month tracking period. The most access English-language pages during the entire Wiki4WorldHearingDay2019 campaign we hearing loss, tinnitus, otitis media, and noise-induced hearing loss. The Portuguese-la guage articles with the highest number of views were hearing and speech therapy, effec of noise pollution on health, solvents, noise-induced hearing loss, and occupational hea On English Wikipedia, the quality assessments are made using a grading scheme aimed to determine the distribution quality of an article on a particular topic within a WikiProject, which is defined as the Wikipedia 1.0 Project. Usually, these quality assessments are carried out by editors who manually tag pages of articles. Then these tags are automatically collected by a bot that generates logs and statistics in the form of matrices of quality or relevance [36,37]. On the Portuguese Wikipedia, an equivalent grading scheme is composed of six indexes (ranging from 1 to 6), with the highest being the best quality rating [38][39][40][41][42][43][44][45].
For an article to be indexed as quality, the algorithm takes into account the semantic structures that indicate quality criteria and possible syntax or editing issues, both conditions affecting the final score [38]. Other criteria such as verifiability and the number of reliable sources determine the outcome [46]. Overall, verifiability, neutral point of view, compliance to the manual of style, presence of sections and images, and completeness are positively rated by the algorithm. The absence of reliable sources, internal or external links, a lack of neutrality (or a disrespect of other content core policies, such as those related to verifiability and original research), and a poor layout negatively affect the final score [46].
Thus, the articlequality model of the ORES UI tool was applied in two circumstances on each page:

1.
To compare the index of quality after the last history revision prior to student activities to the last history revision after the student interventions to find the improved quality.

2.
To obtain the index of quality after the last history revision, conducted right after the end of student activities (for the created/translated content).
In the case of translated pages, only the revision history was used to obtain the quality index. Then comparisons regarding the evolution of the indexes between the two forms of activities (content creation/translation vs. content improvement) were carried out to determine the effectiveness of each approach and its impact.

Global Online Campaigns
Activities from the Wiki4WorldHearingDay2019 campaign were tracked from 21 January 2019 through 31 March 2019. Overall, the results indicated that 74,000 words were contributed to 90 existing and 7 new Wikipedia articles, and 21 images were donated to the open access repository WikiCommons. These expanded (edited) and new articles received more than 2.8 million views during the 2-month tracking period. The most accessed English-language pages during the entire Wiki4WorldHearingDay2019 campaign were hearing loss, tinnitus, otitis media, and noise-induced hearing loss. The Portuguese-language articles with the highest number of views were hearing and speech therapy, effects of noise pollution on health, solvents, noise-induced hearing loss, and occupational hearing loss. Four of these Portuguese-language articles were edited by study participants.
Since the beginning of the subsequent Wiki4YearOfSound2020 campaign on 4 November 2019, overall, more than 1000 articles were created or expanded in several languages, including English, Portuguese, Spanish, Italian, Chinese, and French, in a 12-month period. At the end of 2020, the entire campaign had achieved more than 131 million views for those 1000+ articles. One must note, however, that because pages had been individually edited at different times throughout the campaign, the duration of the tracking period varied with the date of the edit (i.e., articles tracked early in the campaign have longer tracking periods).

Outcomes by the Brazilian Students' Efforts as an Extension Activity: Quantitative Assessment
The student participants were responsible for 60% of the Portuguese-language edits for Wiki4WorldHearingDay2019 campaign and more than 90% of the Portuguese-language edits for the Wiki4YearOfSound2020 campaign. The Audiology Study Group participants' contributions are presented next for each of the global campaigns. The selection of topics and subsequent literature searches were conducted by the students and their project coordinator, focusing on topics that could contribute to the promotion of hearing health. The group expanded preexisting Portuguese-language articles in Wikipedia and translated articles that had been published on the English Wikipedia (see Table 1).
The timeframe for comparing the two versions was 30 days before activities began and 30 days after they ended. Newly created articles were accessed either 30 days after their creation or when they were uploaded to the website (for those who worked on user test pages).
In addition to the contributions shown in Table 1, study participants also performed minor edits, such as adding categories to existing pages. Those edits are important because Wikipedia uses categories to align pages by thematic groups. This enables readers to browse pages related to the same topic, thus consolidating the contents on the platform while broadening users' reading and knowledge scope. For example, the category "audiology" was added to 26 pages, increasing the number of pages with this category by about 33% [47].  (Tables 2 and 3). However, the study group was responsible for 60% of the edits during the Wiki4WorldHearingDay2019 campaign and more than 90% of the edits during the Wiki4YearOfSound2020 campaign on the Portuguese Wikipedia.

Outcomes by the Brazilian Students' Efforts as an Extension Activity: Qualitative Assessment
Tables 4-6 present an overview of the Wiki4YearOfSound2019 and Wiki4YearOfSound2020 campaigns based on the ORES scores for the articles on Portuguese Wikipedia within the contributions of the Audiology Study Group.  [48][49][50]. Tool: ORES UI. Indexes higher than 4 should be considered as an indication of evolution and not actually representative of "good article" and/or "featured article" status, as these statuses are obtained through a community-based decision process.  All scored metrics had relatively higher quality for translated/created articles when compared with preexisting pages. The most frequent ORES indexes (mode) within the articlequality method were 4 for created pages and 2 for preexistent pages. The mean and the median values were about 33% and 75% higher for the created articles compared with the expanded ones, respectively. Beyond that, an improvement was observed in the quality of 58% of the expanded articles.

Discussion
Health promotion has been identified as a strategy to increase reach and service delivery as well as to trigger social engagement [51]. Its goal is to strengthen individuals and communities, increasing their autonomy as users of health services [52]. Wikipedia is considered an important resource for healthcare information in different contexts, one that is able to reach the public, patients, students, and professionals looking for health-related information online. Wikipedia has the mechanisms to respond to highly dynamic situations, to assess the reliability of resources, to invite collaboration, and to leverage expertise; consequently, it is becoming an important platform for health promotion [22,53,54]. Its tools and metrics make it possible to publish content that could promote the uptake of hearing health services, allowing health professionals and the public to evaluate its reach, public interest, and the potential impact of these actions within the community.

Analysis of Reach/Pageviews of the Global Campaigns and the Contributions from Brazilian Students
An objective of the Wikipedia campaigns described in this study was to help the general public in understanding acoustics subtopics such as environmental noise, noise control, hearing and psychoacoustics, speech, and vibration. Another objective was to increase their understanding of the importance of controlling environmental and occupational noise [55].
Undergraduate students of the Speech-Language Pathology and Audiology Program were responsible for 60% of the edits on the platforms during the Wiki4WorldHearingDay2019 campaign and more than 90% of the edits during the Wiki4YearOfSound2020 campaign. Their focused and ongoing participation could motivate other research groups, including Wikipedians who are interested in this topic, and possibly inspire the creation of a WikiProject in Audiology within Portuguese Wikipedia, following the guidelines of other WikiProjects [56][57][58].

Analysis of the Article Quality Assessment of the Brazilian Students' Contributions as an Extension Activity
The number of article visualizations was impressive, indicating both the student's interest in a topic as well as the interest from others who use Wikipedia. This information is valuable to researchers and health and science communicators as it allows them to plan and prioritize their communication efforts. To assess the impact of the activities, an analysis of the quality of the articles (either created or expanded) was carried out. These analyses revealed measurable improvement in the existing articles and good quality in the newly created articles.
The mean metric for the articlequality model for the edited articles (Table 4) rose from 1.83 to 2.5, an increase of more than 36%. The mode value rose from 1 to 2 while the median remained unchanged, suggesting variability among articles. In the articles created or translated through the online campaigns (Table 5), the mean, median, and mode values registered were 3.3, 3.5, and 4, respectively. These scores account for increased rates of 33.2%, 75%, and 100% when compared with those on the other group (Table 6).
Regarding the articles that existed before the current study (Table 4), the lower the previous quality index, the higher the increase registered. While no articles with a previous ORES quality index (articlequality method) of 3 or above had a change in rating after the interventions, most articles with a previous index of 2 registered a 1 point increase, and all articles with a previous score of 1 registered a 1 or 2 points increase. The pages "Tinnitus," "Psychoacoustics," and "Effects of noise pollution on health" had the most progress (+2) in this category. Of the 24 articles from this group, 14 evolved in terms of quality index, although 10 did not register any progress. As for pages that were created or translated from English into Portuguese (Table 5), a mode of 4 was observed. The "Hair cells" and "Hearing protection" pages achieved the highest indexes (of 5), which is equivalent to other pages retaining the "good article" status.
Although article length is often a measure to assess the quality of an article, the results demonstrated that this variable may have little influence on the article quality model predicted by ORES algorithms [59]. Other variables that impact internal quality include the quality of the text's semantics; reliability of references; amount of inner, external, and broken links; existence of multimedia files, and more. These aspects may receive considerably more attention when an article is new [60] (pp. 101-113).

Feasibility of Educational Activities Involving Student Participation in Wikipedia Campaigns
While the approach described in this study is applicable to any field, it seems particularly relevant to the promotion of hearing care, as delay in seeking treatment is seen in several countries, regardless of the differences in availability of healthcare services. This delay is likely to limit the success of many interventions [1][2][3][4][5][6]. While numerous factors can explain why it takes individuals a long time to seek help, researchers argue that many of these factors are influenced by access to information regarding health conditions and the possibilities of care [1][2][3][4][5][6].
In the current study, undergraduate students were able to meaningfully contribute to the efforts of the Wikipedia component of two global health awareness campaigns. Whereas prior studies have used Wikipedia editing in the classroom as a required assignment [61,62], the current experience did not involve credit or grades [63]. Overall, the experience of student editing is described as having a positive impact [62,63].

Study Limitations
This study extended prior findings, but there were limitations inherent in the approach taken. Although students' participation was not factored into grades, students may have felt that their participation would be viewed favorably by the instructor. There are many complexities involved in participatory action research, and we were unable to include them in the present study. This exploratory study, however, revealed several alternatives for follow-up work. The current study was limited in narrowly focusing on Wikipedia platforms rather than broader level technology skills. Future research should examine if this activity has a measurable impact on information literacy, such as if the skills learned through Wikipedia editing carry over to other online environments or influence other online writing, such as blogs and/or discussion forums. In addition, this approach offers opportunities for the role of collaboration in the learning process to be evaluated. Future research could (1) compare individual student work with collaborative editing to determine the effectiveness of each method, (2) strive to collect anonymous data for student feedback, and (3) allow researchers to examine whether Wikipedia activities enhance or limit student learning. An option for large classes would be to assign group work where article feedback would be given to groups rather than individual students. Regarding learning outcomes, we recommend future studies to explore the content of students' edits to determine if the quality of their contributions improved over time.

Conclusions
Technology offers new ways for interacting and acquiring knowledge, which are extremely relevant in the health education field. Currently, skill-and knowledge-based crowdsourcing strategies are important in the promotion of public health. Awareness campaigns, such as those promoted by WHO and Wikipedia, are unique opportunities to improve hearing-related content on a referenced, plain language, open access source, thus providing quality information to the public.
The quality and the amount of coverage of science topics in Wikipedia articles are influenced by the degree in which experts in the field are involved. The expansion and improvement in articles from one subject area usually lead to an increase in pageviews [11,64]. Both approaches-translating content and improving preexistent articles-proved to be equally effective in improving the overall quality of the selected articles. These actions could contribute to the hearing health of the population by providing more freely available online articles containing reliable references as a primary source of information. Furthermore, access to plain language information about a health condition, including its possible diagnosis and treatment, is considered an important tool in increasing the use of health recommendations and services [8].
This educational and health promotion process involved study participants using new strategies to expand the delivery and implementation of scientific content. Reflecting on the study results, which indicated an average improvement of about 36% in the quality scale for the articles, incorporating activities to expand and improve scientific content in Wikipedia are justified both as a hearing health promotion educational experience and as a strategy for knowledge dissemination [65]. Clearly, Wikipedia activities provide a meaningful instructional assignment that can also benefit society. Funding: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Authors conducted this study as part of their employment duties. No other funding was provided.
Institutional Review Board Statement: Ethical review and approval were waived for this study because student class assignments, as a rule, are not systematic data collection efforts intended to develop or contribute to generalizable knowledge and thus do not meet the federal regulatory definition of research [66]. Therefore, as a rule, student class assignments do not fall under the jurisdiction of the Institutional Review Board (IRB) and do not require IRB application, approval, or oversight. Most student class assignments pose little or no risk to students or others. The instructors ensured that students understood any potential for harm and took all possible steps to eliminate the risks to students or individuals outside the class involved in the assignment. Furthermore, in Brazil, studies with these characteristics that rely on data in the public domain are not required to seek approval by an Human Studies and Ethicals Review Board (article 1, sole paragraph, items III and VIII, of the Resolution N. 510, of 7 April 2016 of the National Health Council) [67]. All data and logs are available publicly in Wikimedia sites and can be searched via non-proprietary software maintained by WMF. In addition, according to the Wikimedia Privacy Policy, users are not required to disclose any personal information while registering their accounts on the platform [68], including their own usernames, which do not need to reflect their real names.