Producing an evidence‐based treatment information website in partnership with people affected by multiple sclerosis

Abstract Background and Aims In earlier work, we identified that people affected by multiple sclerosis (MS) can have difficulty finding online treatment information that is up to date, trustworthy, understandable, and applicable to personal circumstances, but does not provoke confusion or negative emotional consequences. The objective was to develop online consumer summaries of MS treatment evidence (derived from Cochrane Reviews) that respond to identified treatment information needs of people affected by MS. Methods A 2‐phase mixed‐methods project, conducted in partnership with consumers and an MS organisation. Phase 1 included review panels with consumers (Australians affected by MS) and health professionals to test paper‐based treatment summaries before development, and pilot testing of the website. Phase 2 involved an online survey after website launch. Results Eighty‐three participants (85% affected by MS) took part. Phase 1 participants strongly endorsed key review summary components, including layering information, and additional sections to aid personal applicability. Participants additionally suggested questions for health professionals. Participants across both phases were receptive to the idea of being provided with Cochrane Review summaries online but were seeking other types of evidence and information, such as personal experiences and the latest experimental treatments, which could not be provided. While the small survey sample size (n = 58) limits application of the results to a broader population, the website was viewed favourably, as a useful, understandable, and trustworthy information source. Conclusion We describe a partnership approach to developing online evidence‐based treatment information, underpinned by an in‐depth understanding of consumers' information needs.


| BACKGROUND
Multiple sclerosis (MS) is a chronic neuro-inflammatory condition with much uncertainty around individual prognosis. Treatments include a diverse array of drugs with some confronting risk-benefit profiles 1 and many novel and experimental treatments. 2 Like many health care consumers, people affected by MS, that is, people with MS and their families and carers, increasingly seek information about such treatments online. 3,4 While most trust the advice of their health professionals, 3 many people affected by MS gain confidence over time to find and interpret online treatment information. 5 Within this context, it is important to provide people affected by MS with up-to-date, reliable, and independent evidence-based treatment information. Systematic reviews, as summaries of multiple studies on a topic, should be the basis for much of the information about benefits and harms. 6 However, previous research has found that consumers, that is, patients, families, carers, and their advocates, can find systematic reviews, and plain language summaries, either difficult to understand or insufficiently detailed to use in decision making. 7,8 They also want to know how research evidence relates to them individually, and determining this can be difficult. 7,9 While the evidence is building on how to best share systematic review evidence with consumers, 7,9,10 including people affected by MS, 11,12 until recently, there has been scant literature or examples about how this translates into an online environment within the context of MS 13 and how to integrate systematic review data with other information needs. 9 Recent developments in consumer-friendly online presentations of benefits and harms include interactive Summary of Findings tables and plain language summaries 14 and the MAGICApp rapid recommendations and decision aids. 15 This paper describes the development of Making Sense of MS Research, 16 an Australian website launched in 2012, that provides summaries of MS treatment evidence from Cochrane Reviews for people affected by MS. We created the website as part of "Integrating and Deriving Evidence, Experiences and Preferences" (IN-DEEP), a mixedmethods study conducted in parallel in Australia 5,17 and Italy. 18,19 Here, we describe the Australian arm of the project.
For the first stage of IN-DEEP, we undertook a qualitative exploration of the online treatment information-seeking experiences, preferences, and needs of 60 Australians with MS and their family members. 5 We identified that people affected by MS can have difficulty finding online treatment information that is up to date, trustworthy, understandable, and applicable to personal circumstances, but does not provoke confusion or negative emotional consequences.
In stage 2, described here, we sought to respond to the unmet treatment information needs identified in stage 1, by developing Making Sense of MS Research. 16 As such, the aim of this project stage was to develop online consumer summaries of MS treatment that were easy to understand and applicable to personal circumstances, met additional information needs, relied on up-to-date and trustworthy sources, and mitigated confusion and negative emotional consequences.

| METHODS
The methods for this project were informed by the principles of knowledge translation 6 and consumer participation in research. 20 In knowledge translation, the focus is not just on the content and message being shared but is also founded on an understanding of the target audience and the broader context. 21 Involving consumers in research has an intrinsic value of encouraging democracy, accountability, and transparency and an extrinsic value of improving the quality, impact, and relevance of the research produced. 22,23 In practice, both involve interactive processes that share research between its creators and users, with varied activities and multiple iterations. 6 We describe this study in 2 phases. Phase 1-developing the website-involved the development of paper-based treatment summaries that were tested in review panels involving people affected by MS and others, followed by website creation and pilot testing. The purpose of these review panels and website pilot testing was to help inform the website content and design the information presentation format.
Phase 2-website feedback survey-occurred after the website was launched and involved an online survey of website visitors. The purpose of the evaluation of the consumer summaries and the website was to gather user feedback about the presentation and usefulness of the consumer summaries and about website navigation. Reflecting commonly used terminology in Australia, in this paper, we use the term "consumer," which means patients, families, carers, their advocates, and representatives, interchangeably with "people affected by MS."

| Project team, and consumer and operational advisory groups
The project team included Australian (SH, AS, MH, and RO) and Italian researchers (GF, CC, and PM), and staff from Australian and local (Victorian) MS societies (MPS and SS). We also sought input from 2 advisory groups convened for the project: a consumer advisory group and an operational advisory group. The consumer advisory group (CLC, RS, and CAM) included 2 people with MS and an MS health information specialist. They prioritised the Cochrane Reviews to be included on the website and provided advice on data collection and analysis, and input into website layout, content, and wording. 24 The operational advisory group included clinical managers (n = 2), IT staff (n = 2), and communications personnel (n = 2) from the local Australian state-based MS organisation. They played a major role in the design and technical aspects of the website and reviewed content from a clinical perspective.

| Phase 1: developing the website
The development of the treatment summaries from the Cochrane Reviews, along with all additional website content, and overarching structure and format, was informed by existing literature and best practice documents. There were also shaped by the findings of the previous IN-DEEP study (stage 1), 5 together with input from people with MS, and others, described in here (IN-DEEP stage 2).

| Developing paper-based summaries
We selected 2 Cochrane Reviews of MS treatments (prioritised by the consumer advisory group) and developed them into paper-based treatment summaries for feedback by the review panels. We chose these reviews (one on a medication used by many people affected by MS 25 and the other about exercise 26 ) because of the relevance of this information to many people and because of the different types of data to be summarised. We followed research-based principles for presenting treatment information to consumers, such as layering, 27 and used a combination of words, numbers and pictures, and absolute, rather than relative, frequencies. 28,29 The information we presented differed from the usual format of a Cochrane Review in the following ways: It was significantly shorter and focused on key information only (ie, inclusion criteria and results); plain language was used throughout; graphical illustrations of numerical information were used; and novel sections, explaining how results applied to individuals and questions that consumers can ask their health professionals, were included.

| Review panels
The aim of the review panels was to seek formal feedback on the paperbased treatment summaries before they were transferred to an online format. Recruitment involved purposive sampling from participants of the first stage of the IN-DEEP project 5 and from the networks of the project team and the MS organisation. We sought to include people affected by MS, clinicians, and staff of the MS organisation.
One week prior to the review panels, we mailed participants a pack containing a document explaining the nature of the project, the 2 treatment summary templates, and information and consent forms.
We included a one-page feedback form, whereby participants were encouraged to record their immediate impressions of the templates and bring this to the panel discussion.
We held 2 review panels (a form of group interview) 30,31 at a local MS organisation office, with each panel session lasting 2 hours. Two researchers were involved as facilitator (SH) and note-taker (AS). The following 5 questions, based on those used in the DECIDE project (work package one), 32 were used to guide the discussion: "do you understand the information presented?"; "is it helpful to you?"; "is there anything missing?"; "is there anything superfluous?"; and "do you have any suggestions for improvement?" One researcher (AS) collated participants' feedback sheets, the researchers' detailed written notes, and audio recordings of the review panel sessions, which were reviewed by a second researcher (SH). We used the audio recordings to supplement the written notes. One researcher (AS) grouped all participant feedback under 1 of 5 categories: one overarching category on content, formatting, or general feedback, and 4 specific categories corresponding to the 4 main sections of the treatment summary template (ie, "The short answer," "The detailed answer," "What does this mean for me?" and "Using this information"). Within each of these categories, individual feedback items were grouped according to "likes," "dislikes," and "suggestions." We considered the implications of each individual feedback item, culminating in a list of recommendations for changes and/or additions to the individual review templates or, more broadly, for the website.
We subsequently discussed and agreed on the recommendations with the project team and consumer advisory group.

| Developing the pilot website
We created a pilot website after integrating feedback from the review panels. To guide the website layout and format, we used the HONcode principles 33 and the Harvard School of Public Health Guidelines. 34 Web design and functionality were informed by the Web Accessibility Initiative Guidelines, 35 as well as by input from the consumer advisory group. To cater for people who did not have access to the Internet and to enable website visitors to discuss the treatment summaries with their health professional, we created downloadable PDF versions of the treatment summaries. MS organisation clinical managers and an MS organisation librarian reviewed the content for relevance to the Australian context and for readability.

| Website pilot testing
After the development of the pilot website, we invited people affected by MS to take part in pilot testing. Participants were recruited from people who took part in the review panels and through the networks of the project team.
We asked pilot testing participants to visit the website and complete an online survey (see Supporting Information). Participants subsequently took part in a 30-minute telephone interview to discuss their survey responses with a researcher who had not been involved in the development of the website (MH). In response to the survey data and interview responses, we made minor changes to the website, such as incorporating additional links to further information and reordering of the site page tabs.

| Ethical approval and informed consent
Participants in all phases provided informed consent to participate, via a written or online form. We received approval from the Human Ethics Committee, Faculty of Health Sciences, La Trobe University (numbers 11-169 and 12-128). We offered review panel participants a $50 voucher for their participation. We did not offer reimbursement to participants in other project phases.

| Participants
A total of 83 participants took part of the project, participating in review panels (n = 16), website pilot testing (n = 9), or by completing the website feedback survey (n = 58) (see Table 1). Of the 61 participants who provided demographic information, the majority were people with MS (75%) or family members of people with MS (12%).
Over two-thirds of participants were female (71%) and aged between

| Easy to understand
To present the treatment information in a format that was straightforward and clear, we created 2 sections (the "short answer" and the "detailed answer") to summarise the benefits and harms of the treatment, so that the information was effectively layered. 27 In "the short answer" section, we provided a one-paragraph summary of the Cochrane Review results. In "the detailed answer" section, a combination of words, numbers, and graphical images was used to describe the effect of the intervention compared with placebo, on the main outcomes collected in each review, thereby presenting the same review findings in 3 different formats 28,29 (see Figure 1). We expressed absolute numbers as natural frequencies (ie, 5 of 100).
Review panel participants endorsed the concept of layering the information and providing detailed scientific information in a combination of formats. Some of the nonconsumer participants were concerned about consumers' abilities to make sense of the information, but this was not reflected by consumer participants. There were mixed responses to the 100 smiley faces graphic that we used in the review panel sessions (see Kasper's multifigure pictographs 12 for an example).
After consultation with colleagues, a new horizontal comparison graphic was used on the website instead (see Figure 1). Several participants suggested that the addition of a hyperlinked glossary would be preferable to trying to explain scientific terms in the text. The glossary was subsequently created for the website.

| Applicable to personal circumstances
To provide treatment information that was more personally applicable, we created an additional section, specific to each treatment summary, called "Does this apply to me?" There, we described the characteristics of the people included in the studies and the parameters of the interventions studied therein. 9 We also added a section called "Questions for my health professional" that was specific to each review. 33 We developed a set of questions, in consultation with the consumer advisory group, that consumers could use as a prompt for discussion with their health professionals. Example questions included "How soon after diagnosis is it recommended to take interferons?" and "What kind of rehabilitation would be right for me?" Both the "Does this apply to me?" and "Questions for my health professional" sections were strongly endorsed in the review panels as ways to help readers apply the information to their specific situation.

| Meets additional information needs
To acknowledge the additional information needs of consumers, we created a section called "Find out more," with the aim of connecting readers to their local MS organisation and providing links to other online information and evidence sources (selected in consultation with MS organisation staff). 9,33 This was one of the most strongly endorsed concepts in the review panels and the pilot testing, but many consumers wanted more information and links than those provided. Some

| Trustworthy
At the review panels, there were questions about Cochrane and how Cochrane Reviews were created. For example, some participants asked FIGURE 1 Screenshot showing presentation of the evidence in "the detailed answer" SYNNOT ET AL.
if Cochrane was trustworthy and if the information was up to date. In response to this, we created a Frequently Asked Questions page, drawn from some of the questions posed at the review panels (for example, "What is the Cochrane Collaboration?" and "Why don't the treatment summaries always give a clear answer?"), and a page called "About

| Mitigates confusion and negative emotional consequences
To mitigate confusion or unnecessarily negative emotional consequences, consumer advisory group members reviewed the language used in the summaries, to ensure that it was appropriate to a broad range of consumers. We also added a question to the Frequently Asked Questions section to offer reasons as to why the review summaries could not always provide clear answers. To mitigate the risk of confusion or concern for website readers, we included questions such as "Why do these numbers look different from what I have seen elsewhere?" The website encouraged readers to contact their local MS organisation if they were concerned about conflicting information.

| Phase 2: website feedback survey
Once launched, the website received 3873 visits (75% of which were unique visits) in the 2-month period during which the feedback survey was live. During this evaluation period, 61 people elected to respond to the survey, with 58 useable responses included in the analysis (responses from 3 people were too incomplete to include). Given we did not actively recruit participants to the survey, but instead recruited them to view the website, the small sample size is not unexpected but does limit interpretation of the usefulness of the site to a broader population. However, of those who completed the survey, nearly 80% (n = 46) responded positively to the research summaries and the website as a whole. As well as quantitative data, respondents gave free-text responses that indicated they found the information and website layout to be clear, aesthetically pleasing, and easy to navigate and understand. Overall, the presentation of the benefits and harms of each intervention was well received, with more than 75% of respondents agreeing or strongly agreeing with questions about understanding benefits and risks (see Supporting Information, S2 Table).
Other questions were designed to probe reflections about personal applicability ("The website has helped me to understand how the treatments might be useful for me"), trustworthiness

| DISCUSSION
In this study, we sought to address the unmet treatment information While Cochrane Reviews remain the gold standard in systematic reviews of treatment research, 45 and systematic reviews should form the basis of evidence-based treatment information shared with consumers, 6  Our findings affirm the importance of partnerships between evidence producers and trusted information providers, such as peak bodies and member organisations. Not only do such organisations provide some of the additional information that consumers seek, but also groups like MS societies are considered trustworthy sources of health information. 3 This is important within the context of Cochrane reviews, given they were unfamiliar to consumers in our study. This unfamiliarity has been found by others to result in scepticism about the trustworthiness of information based on Cochrane Reviews. 9,49 Additionally, presenting the evidence from systematic reviews is a science in itself. Another relevant development is a broadening of systematic review approaches, including network meta-analysis (where multiple interventions can be compared head-to-head), 56 mixed-method reviews (where quantitative data are synthesized with qualitative experience data), 57 and reviews of observational data about side effects. 58 These types of reviews open the door for evidence summaries to better meet consumer needs, by presenting empirically derived comparative effectiveness, longer-term data on potential harms, and the lived experience of others.
We acknowledge a number of limitations. First is the small number of participants who completed the online survey, limiting application of the evaluation results to a broader population. The low response rate to the survey may be due, in part, to dissemination or recruitment shortcomings: We were reliant on our project partners for website promotion, and we elected to promote the website, rather than explicitly recruit people to complete the survey. Alternatively, those who did not find the website helpful may not have chosen to complete the survey, potentially biasing results towards more favourable responses.
Second, it is unclear how well the website caters to those with low literacy levels. The educational background of consumers and family members involved in the development was high (mainly university educated), and we did not use a standardised tool (eg, Flesch-Kincaid Grade) to assess the readability of the website. 59 However, the fact that we received largely positive feedback from survey responses, including from those who did have limited levels of education, suggests we may have met, at least, some of the information needs of people with limited literacy. A final limitation was that the assessment of important concepts, such as understanding, were self-reported, rather than objectively measured. A more objective measure of understanding is likely to have generated more modest results, 12 although this was not within the scope of this project.

| CONCLUSION
This paper describes an approach to developing evidence-based treatment information that is underpinned by in-depth understanding of the information needs of consumers. This understanding was facilitated by the research team's ongoing partnership with consumers, throughout the development of the Cochrane Review information summaries, and the website that hosts them. The evaluation indicates that some of the unmet treatment information needs of people with MS were met through this novel partnership approach. Equally, we have demonstrated that consumers seek more than just the information contained in Cochrane Reviews to help them understand and apply treatment information. Future projects that include partnership with consumers, as well as with peak health organisations and other treatment information providers, may offer opportunities to fulfil these additional consumer treatment information needs that extend beyond treatment risks and benefits.