Interventions to Assist Health Consumers to Find Reliable Online Health Information: A Comprehensive Review

Background Health information on the Internet is ubiquitous, and its use by health consumers prevalent. Finding and understanding relevant online health information, and determining content reliability, pose real challenges for many health consumers. Purpose To identify the types of interventions that have been implemented to assist health consumers to find reliable online health information, and where possible, describe and compare the types of outcomes studied. Data Sources PubMed, PsycINFO, CINAHL Plus and Cochrane Library databases; WorldCat and Scirus ‘gray literature’ search engines; and manual review of reference lists of selected publications. Study Selection Publications were selected by firstly screening title, abstract, and then full text. Data Extraction Seven publications met the inclusion criteria, and were summarized in a data extraction form. The form incorporated the PICOS (Population Intervention Comparators Outcomes and Study Design) Model. Two eligible gray literature papers were also reported. Data Synthesis Relevant data from included studies were tabulated to enable descriptive comparison. A brief critique of each study was included in the tables. This review was unable to follow systematic review methods due to the paucity of research and humanistic interventions reported. Limitations While extensive, the gray literature search may have had limited reach in some countries. The paucity of research on this topic limits conclusions that may be drawn. Conclusions The few eligible studies predominantly adopted a didactic approach to assisting health consumers, whereby consumers were either taught how to find credible websites, or how to use the Internet. Common types of outcomes studied include knowledge and skills pertaining to Internet use and searching for reliable health information. These outcomes were predominantly self-assessed by participants. There is potential for further research to explore other avenues for assisting health consumers to find reliable online health information, and to assess outcomes via objective measures.

While the role of health professionals in advising and assisting health consumers with decision making is undisputed [1,16,17], it is unrealistic, in terms of the health professional's time and ability, to control out-of-consultation behavior of consumers, and to expect consumers to rely entirely on written and verbal information provided by their regular practitioner(s). Indeed, health consumers empowered by information-seeking online may be more engaged in the management of their health conditions, and form more productive relationships with their healthcare practitioners [18].
In order to effectively engage in self-management, health consumers must be able to effectively find, understand and utilize relevant health information [19][20][21]. At the same time, health consumers must also be able to discern reliable from less-reliable information. Collectively, these abilities and skills constitute the definition of an individual's health literacy [19][20][21]. The literature suggests that many health consumers have low levels of health literacy [21][22][23][24][25][26] which hinders their ability to effectively find, understand and use reliable health information to assist them manage their health conditions. As use of the Internet requires technological knowledge and skills [9], finding health information online potentially becomes an even greater burden on consumers with poor computer literacy as well as low levels of health literacy.

Rationale and Objectives
Given the prevalence of chronic diseases in society and the important role self-management plays in chronic disease management, there appears to be a need for initiatives to assist health consumers to develop their capacity to find reliable health information on the Internet, thereby potentially contributing to improved levels of health literacy. Our comprehensive review follows a systematic review by Car et al. [9] that examined the effects of online health literacy training interventions on health outcomes. We intended to focus on the gambit of interventions (face-to-face, online or via other means) implemented by researchers to assist adult health consumers to autonomously find reliable online information about chronic health conditions, and where possible, their outcome measures. This includes, but is not limited to, training interventions.
As the focus of our review is on identifying and descriptively comparing humanistic interventions, as opposed to evaluating formal clinical trials, assessment of the quality of each study (as would be performed in a systematic review) was not formally conducted. Nevertheless, we adopted a systematic approach to the identification and selection of relevant studies, and applied methodological [27] and reporting [28] standards for the conduct of systematic reviews where practical.

Methods
The method was guided by the Methodological Standards for the Conduct of Cochrane Intervention Reviews [27], where relevant, recognizing the humanistic nature of the interventions of interest.

Eligibility Criteria
The eligibility criteria for published studies were developed via consultation between all authors of this review, and were: 1. Participants: at least 18 years of age 2. Intervention: any approach where the intention, primary or other, was to assist health consumers in autonomously finding existing reliable online information related to chronic health conditions 3. Language: publications in English.
Publications were included if the study, or a portion of the study, met all of the above criteria. Studies that met criteria 2 and 3, but included some participants less than 18 years of age, were considered relevant. Studies were excluded if the primary intervention involved the development of new online health information material, such as producing easy-to-understand material for the purpose of assisting people with low health literacy. Studies were also excluded where assistance was provided to help participants use a particular existing online health website. While the reporting of outcomes for studies is a critical component to evaluating quality, and is desirable, studies that did not report outcomes were not excluded from this review as the primary objective of this review was to identify the types of interventions that have been implemented.

Information Sources
Four health-related databases and two popular search engines for 'gray literature' were utilized for this review. While the term 'gray literature' is poorly defined in literature, we defined it as studies published in a format other than academic journals. Reference lists of selected publications were also searched, and authors were contacted where clarification or more information was needed for data analysis.
The databases used were PubMed, PsycINFO, CINAHL Plus and Cochrane Library. Gray literature was searched via WorldCat and Scirus. No limits were applied to dates, although it was expected that most research in this field would be recent due to the relative recency of the Internet and Information Technologies. The last search was conducted on 11 February 2013.

Search
Search terms and search strings were agreed upon by all authors of this review, and further refined by the first author, KL, after pilot searching various permutations of search terms in PubMed and CINAHL Plus. To optimize efficiency and minimize the volume of irrelevant publications returned, a single search string was used for all databases and gray literature search engines. A second search, using MeSH terms, was conducted on the four databases. This is because MeSH terms are manually assigned after a study has been published; thus, a search using MeSH terms could potentially produced different results. Manual searching of reference lists from selected publications were subsequently conducted to ensure a more comprehensive search.
An example of a search string that was used for this review is: ((consumer* or patient*)) AND ((find* OR search* OR navigat* OR seek* OR access* OR retriev* OR locat* OR identif*)) AND ''health literacy'' AND informat* AND ((internet or online or web*))

Study Selection, Data Collection Process and Data Items
After retrieval of publications from the aforementioned databases and search engines, duplicate publications were removed.
A stepwise approach of selecting relevant publications firstly via title, then abstract, then full text was used, adapting relevant sections of the PRISMA flow diagram for reporting of systematic reviews [28]. In this approach, titles of publications were firstly screened for relevance to the review, with relevant publications retained. Where relevance was unclear from examination of title alone, the abstract was then scanned and irrelevant publications were subsequently removed. Publications in which both title and abstract were examined, and relevance to the review was still unclear, were retained for full-text review. Reference lists of selected publications were manually searched for further relevant sources.
Eligibility of publications was further ascertained by full-text review. Authors were contacted for supplementary information where necessary.
A data extraction form was developed to assist data collection and subsequent analysis. This form was developed and piloted by KL, and reviewed by LE. All elements of the PICOS model [28] were incorporated into the form. As few relevant papers were anticipated, trial of the form was not considered necessary, although refinements were incorporated during data extraction. KL extracted data from the included studies, and LE, confirmed the extracted data. Consensus was reached on all points via discussion between KL and LE.
As this review primarily focuses on identifying types of interventions with a view to also identifying types of outcomes where possible, as opposed to examining the effects of these interventions on outcomes, risk of bias in individual studies and across studies was not assessed. It was anticipated that little methodological comparison would be viable.

Study Selection
The literature search identified 707 publications after duplicates were removed (Figure 1). Of these, 680 publications were excluded through the screening process (via their titles and abstracts). Common reasons for exclusion were that the publications reported train-the-trainer interventions with no follow-up of the trainers subsequently coaching health consumers, interventions to help consumers find information on a particular website, studies about the various implications of poor health literacy, and exploratory studies of health information-seeking behaviors.
A total of 27 publications were reviewed in full to assess their eligibility for this review. To further assist in assessing eligibility of publications, 14 authors were contacted for supplementary information, with four replies. Overall, seven journal publications and two gray literature reports were deemed eligible for synthesis and analysis. Due to missing data in the gray literature reports, they are described in a separate section from the published studies.
The interventions for three [30,31,34] of the seven studies were administered in a single session. The total duration of studies with multiple-session interventions ranged from three weeks to nine months.

Design of Interventions
Interactive workshops. Interactive workshops featured as the main intervention in five [30][31][32]34,35] of the seven published studies. The style of the interactive workshop for one study [31] was unclear, however, four studies [30,32,34,35] used a combination of a didactic approach to teaching and hands-on activities that involved participants searching for health information online. In most of the studies, participants were issued a copy of the presentation slides, as well as lists of credible health websites. Further, discussion amongst participants was encouraged during the workshops in three studies [30,32,35]. In particular, collaborative learning was emphasized in one [35] of the studies. Although the intervention for this study included a didactic component, the workshop trainer encouraged interaction between participants. In another study [31], a website hosted the online version of the in-person interactive workshop. This online version used the same content as the in-person workshop.
Common workshop topics included how to judge reliability of health information and credibility of health websites, and awareness of credible websites. Two studies [34,35] used content from The Trainer's Toolkit from the NIHSeniorHealth website (http://nihseniorhealth.gov/toolkit/toolkit.html) as the foundation of their workshops. Participants in one study [31] were provided information on their medical condition, in addition to being educated on how to find credible health information online.
Health literacy curriculum and community outreach. In one study [33], a health literacy curriculum was designed and trialed in two middle schools, two high schools and one adult education program, delivered by teachers and librarians. The curriculum was designed as five one-hour lessons with activities, although teachers and librarians were allowed to deliver each lesson at their own pace. As a result, the duration of the intervention ranged from three weeks to three months. In the final lesson, participating students shared what they had learnt with seniors in their community. In this way, seniors were taught by participating students how to find health information online.
The curriculum established that health information is commonly organized via disease type and population, and participants practiced search techniques. Students were also given a checklist adapted from the QUICK website (http://www.asparis.net/ lowerschool/quickreview/www.quick.org.uk/menu.htm) for evaluating the credibility and reliability of online health information.
Online portal with support via videoconferencing. One study [29] took an arguably more holistic approach, in the development of an online portal to house three modules: selfmanagement, health education, and social networking. The selfmanagement module provided a space for each participant to access their individualized care plans. These plans were reviewed during the bi-weekly videoconferencing with a telehealth nurse, and were amended, if needed, by the participant's physician. During these videoconferences, participants were also encouraged to raise questions with the telehealth nurse. The health education module consisted of age and culturally-appropriate educational videos and links to various health-related websites. The social networking module provided a space for participants in the study to interact with one another, and participants were encouraged to share preferred educational resources.

Outcomes Studied
All seven published studies demonstrated either positivesignificant, or positive-but-non-significant outcomes in at least some of the measured outcomes (Table 1). No study reported any worsening of outcomes from baseline/pre-intervention. Of the measured outcomes, many were self-reported. Two of the published studies [29,31] included some form of objective measure -one study [29] used biometric and biochemical markers, another study [31] used an objective test of knowledge. One study [32] conducted a follow-up assessment up to nine months postintervention.
Knowledge and skills. The breadth of knowledge and skilloriented outcomes included participants' knowledge of their medical condition [29], awareness of what is evidence-based health information [30,33], knowledge of credible online health resources [31], ability to find reliable online health information [30], ability to find relevant online health information [34,35], ability to evaluate reliability of online health information [32,33], and general computer/Internet knowledge and skills [35]. The majority of the knowledge outcomes were based on self-perceived measures of knowledge, with the exception of two studies [29,31]. Carter et al. [29] applied a brief test of diabetes knowledge as part of a survey pre-and post-intervention, whereas Gross et al. [31]   measured knowledge of credible online health resources via a pretest and post-test questionnaire based on items covered in the interactive workshop. Two studies tested their participants' skills: one [32] tested participants' ability to evaluate reliability of online health information by instructing participants to rate two preselected websites on five dimensions of website quality, as specified by a previous study conducted by Kim et al. [36]; while the other [34] investigated participants' ability to find relevant online health information by instructing participants to find answers to two questions on a pre-selected health condition. Attitude and behavior. In one study [30], the key attitude and behavior-oriented outcome were participants' change in the way they search for health information, and in another study [32], information-seeking self-efficacy. In both of these studies, attitudes and behaviors were assessed based on self-perceived measures.
Health outcomes. One study [29] indirectly assessed health outcomes in patients with diabetes via measurement of biochemical and biometric markers for both treatment and control groups pre-and post-intervention: mean weight, mean blood pressure, and mean hemoglobin A1c.

Gray Literature Reports
The interventions, outcomes and brief critique of relevant aspects of the two gray literature reports are summarized in Table 2. Both reports relate to projects aimed to facilitate access to reliable health information for various communities in the United States. The Access to Resources for Community Health (ARCH) project The use of group training sessions was common to both projects. The ARCH training focused on teaching general computer and Internet skills, and how to use their project's website to find reliable health information. The Medline in the Mountains training focused on teaching skills to evaluate reliability of health information, and how to use health information databases; their website appeared to supplement the training, as it hosted materials on how to evaluate reliability of health information, and listed various health information databases. While both projects were conducted in the United States, the ARCH project was conducted in a metropolitan city, while the Medline in the Mountains initiative addressed various geographically isolated and rural communities. The ARCH project utilized focus groups to evaluate their outcomes, while a pre-/post-intervention survey was utilized in the Medline in the Mountains study, albeit with limited data reported.

Discussion
The seven published studies [29][30][31][32][33][34][35] included in this review presented a number of substantive design limitations, including small samples and the use of descriptive analysis. Outcomes were predominantly assessed via self-reported pre-post measures, which arguably have greater potential for bias [37][38][39]. The majority of these studies did not describe any validation process for the instruments used. Further, only two studies [29,32] were randomized-controlled studies. Despite this, four [30,31,33,35] of the five remaining studies assessed either participants' attitudes or outcomes based on pre-and post-intervention measures; this facilitates a more meaningful way to compare the impact of the intervention. Nevertheless, only two [30,35] of these four studies went beyond the use of descriptive statistics to assess statistical significance of their findings. Thus, the design characteristics, analysis, and perceived overall quality of these studies highlight areas to be addressed in future research. Furthermore, the nature of the studies did not lend themselves to being evaluated systematically.
Our review of the intervention type identified that five [30][31][32]34,35] studies used workshops with varying levels of interactivity. Although a didactic approach to helping participants search for online health information was not the sole approach to teaching participants, it was a prominent approach that featured in all of the included studies. Didacticism is a prominent, yet arguably outdated, philosophy in education [40]. Despite vast literature suggesting that student-centered pedagogical approaches may be more effective than didactic education in formal education settings [40][41][42], such approaches do not appear prominent in the education of health consumers, as based on our findings, and it is unclear whether people with limited health literacy respond better to, or prefer, didactic approaches to learning. The concept of patient-centered care -whereby patients have active roles in healthcare, and healthcare is based on the individual patient -is flourishing [43][44]. It appears that a didactic approach to developing health consumers' online information-seeking skills may only partially address the issue, given factors such as the recognized importance health consumers' play in day-to-day management of their health, and an era of fast-developing technologies and information abundance. Thus, there appears to be potential to apply various student-centered pedagogical approaches to educating health consumers. Further, it can be argued that a potential contributing factor to the positive results demonstrated by the five workshop-based studies is due to engagement [45] between the workshop trainer and the participants. Thus, reproducibility of results could be problematic.
It is also important to note that, of the included studies, only one study [35] explicitly indicated the use of a particular learning framework -collaborative learning. There is therefore potential for future studies to examine different learning frameworks in the online health literacy space.
All of the seven studies measured varying facets of knowledge, skills or both domains; in a majority, knowledge and skill domains were assessed via measures that were self-reported by participants.
While it is important to gain insight into participants' perceptions of the impact of the intervention, the use of objective tests would arguably improve objectivity and minimize potential for bias. Despite the apparent lack of objective tests in many of the included studies, it is important to note that there were no reported worsening of outcomes from baseline/pre-intervention in any of the included studies. This suggests that, while the effectiveness of didactic approaches are yet to be compared with other approaches, current identified interventions do not appear to have negative or detrimental outcomes.
In the two included gray literature reports, a didactic approach to helping participants search for reliable online health information was prominent -similar to the findings from the seven included published studies. The intervention for the first of these studies aimed to general computer and Internet skills, with a view to optimize participants' use of a particular website developed specifically for the project. The second study focused on improving skills to evaluate reliability of health information and use of health information databases. These interventions also bear similarity to some of the included published studies. In terms of evaluation of outcomes, no objective measures were used.

Limitations
Few studies were deemed eligible for inclusion. It is possible that the use of additional search strings may yield more relevant studies. While the search strings were reviewed by all authors of this review (discipline experts), and advice had been sought from a medical librarian prior to the review, participation in the searches by a medical librarian may have provided a different perspective. While extensive, the gray literature search may have had limited reach in some countries. The finding of few relevant published studies on this topic highlights the potential for further research. Furthermore, their varied application of scientific methods limited our ability to compare them systematically.

Conclusions
Despite the increasing pervasiveness of, and reliance on, the Internet in the area of health information, we found few reports of interventions to assist health consumers to find reliable health information online. The identified studies measured varying facets of knowledge and/or skills, and commonly reported a didactic approach to training their participants. Outcomes studied were predominantly assessed via self-report by participants, and while arguably subjective, were largely positive and there was no report of worsening of outcomes. As such, there is considerable scope for further research to enhance consumer health literacy in the context of sourcing online information. With the pervasion of mobile technologies and popularity of social media, future research could focus on these developments. Specific interventions could focus on transparent labeling of trustworthy websites -or conversely, 'blacklists' of websites deemed biased or otherwise unreliable.
Additionally, as only one identified study explicitly indicated the use of a particular learning framework, there is also potential for further research to explore the use of various learning frameworks in the online health literacy space.