An infodemiology study on exploring the quality and reliability of colorectal cancer immunotherapy information

Background Immunotherapy is a new treatment modality which promises hope for advanced colorectal cancer patients. To our knowledge, no previous studies have evaluated the quality of patient information available on this topic online. Objective This study will explore the quality and reliability of colorectal cancer immunotherapy information using the Journal of the American Medical Association (JAMA) and DISCERN tools. Methods Design thinking methodology was integrated with systematic scoping reviews framework to inform our descriptive observational media analysis study. Google Chrome was used to run four searches using prespecified search terms selected according to the top patient concerns about immunotherapy. The first 20 relevant results were identified (n = 80) and then duplicates were removed. Descriptive narrative univariate and bivariate analysis was done for the relevant variables. Results The total of included websites was 17. Most websites score <3 points on JAMA and fair/poor on DISCERN. Most of the websites that score ≥3 points on JAMA and excellent/good on DISCERN have a charity affiliation. A total of 58.8% of the websites present the date, 41.2% demonstrate authorship, and sources are mentioned in 29.4% of the websites. Lack of content was noticed in providing the prognosis of patients if no treatment is given, clear aim and the effect of treatments on patient's quality of life. Conclusion Assessing the reliability of information about cancer treatments online remains a challenge. Further research is required to understand the patient perceptions and use of online information and whether it has an impact on their behavioural health outcomes.


Introduction
Colorectal cancer is the second most common cancer in women and the third in men. 1 In Ireland, colorectal cancer is the second commonly diagnosed and fatal cancer as there are 2400 new cases detected annually contributing to 11% of cancer deaths. 2 The atypical clinical presentation of colorectal cancer symptoms in the early stages of the disease along with underuse of screening tests leads to delay in its diagnosis and treatment. 3olorectal cancer therapy is a comprehensive multidisciplinary process.It includes surgery, physical (radiotherapy), chemical (chemotherapy), biological (immunotherapy) and psychological therapy.Although surgery succeeds to clear the tumour in 80% of patients, 50% have micro-metastasis at the time of surgery.Chemotherapy is approved as an effective treatment modality for local metastasis but shows only modest efficiency against distant metastasis.Patients with distant metastasis have a 12% survival rate, and immunotherapy offers hope to patients with metastatic colorectal cancer. 4 distinguishing feature of metastatic colorectal cancer is its microsatellite stability status, which a biopsy measures.It can present as microsatellite stable, microsatellite instability-low and microsatellite instability-high and is used to match patients to immunotherapies. 3,5mmune checkpoint inhibitors immunotherapies specifically with monoclonal antibodies show promising outcomes in patients with microsatellite instability-high colorectal cancer and include pembrolizumab, nivolumab and ipilimumab. 6Beyond monoclonal antibodies, monospecific and bispecific antibodies, cellular therapies, vaccines and cytokines targeting other immune checkpoint molecules, macrophages and other components of innate immunity are under clinical trials for colorectal cancer patients. 3uring the era of patient-centred care, cancer patients are becoming active participants in treatment-related decisions and information seeking has been demonstrated to play a critical role in helping patients cope with the uncertainties associated with cancer diagnosis and navigating treatment. 7 study done by Chua et al. 8 (2018) found that cancer patients rely on the internet for obtaining information, focusing their search mainly on cancer diagnosis and treatment (90%), then side effects of treatments (87%), followed by complementary and alternative therapy (70%), depending on several factors.
][10][11][12] It has been established in a study done in Singapore that female patients, those of a higher education standard and younger patients prefer to search for information online. 8Health status has also been found to be a significant factor in online health seeking. 8,13Houston and Allison 14 reported that those who disclosed poor health status were more frequent users of the internet for health information.Internet users with chronic illnesses and comorbidities were noticeably more likely to use the internet as a source of information on health. 13Colorectal cancer patients were reported as less information seekers than patients with breast and prostate cancer, and they had concerns regarding credibility of the information available online. 9,15 study done by Fogg et al. 16 (2003) explored that most consumers of web-based information depend on superficial and quick methods for credibility evaluation.They rely on site presentation and visual design rather than considering content and source presentation. 17This was also explained by Flanagin and Metzger as they found that web users try to use less mental efforts to judge reliability of online information.Finding high-quality sites is challenging, and there are problems with author identification, completeness and accuracy of the information provided on treatments. 18ith the bombardment of pharmaceutical advertisements and media coverage, in conjunction with the undesirable side effects of treatments, understanding website quality is useful to inform educational initiatives to improve patient's digital literacy and use of online information during the cancer journey. 7Also, as immunotherapy is relatively new, no previous studies have evaluated the quality of information available on this topic for patients online.Thus, the health belief model was applied in our investigation as a concept to explain how individual behaviours can affect seeking and adherence to treatments and impact their physical health outcomes. 19

Aim and objectives
The aim of our study is to evaluate the quality and reliability of colorectal cancer immunotherapy information on patientoriented websites using the Journal of American Medical Association (JAMA) benchmark and DISCERN tools.The objectives are as follows: 1. Categorise the quality of website's information using JAMA and DISCERN tools and describe each category by website's affiliation type.2. Describe the website's information reliability by reporting their authorship, attribution of references, currency of the date content and disclosure/ownership of websites.3. Describe the website's information reliability by reporting their aim, source and date of the information, knowledge uncertainty and bias of the information provided and how good the quality of information about treatment choices.

Methodology
A human-centred, design thinking methodology approach was used in our study, and it includes the following steps in order: empathy, problem definition, ideation and designing a prototype and test its applicability. 20The first two steps were selected for application in our investigation to identify issues related to online information about immunotherapy from a patient perspective.The empathy stage was applied when conducting the search strategy, and problems were identified during the analysis and presented as key findings.This methodology was integrated with Arksey and O'Malley's (2005) methodological framework of systematic scoping review for conducting our descriptive observational media analysis study. 21

Identify research question
The Population Concept and Context framework was used to develop the research question. 21The population consists of colorectal cancer patients, their carer's and the public.
The concept is the quality and reliability of colorectal cancer immunotherapy information.The context is the online patient-oriented websites.So, the research question is 'What is the quality and reliability of colorectal cancer immunotherapy information on patient-oriented websites?'

Search strategy
A literature review research technique was used to understand the context of immunotherapy and to have an insight of user's concerns, 22 barriers 15 and needs 23 (Figure 1, Appendix I).Search terms were selected according to the top patient concerns about immunotherapy. 22n 17 July 2021 at 9:00 am ECT, a clean-installed Google web browser was used to conduct the search using different search terms 24 namely, 'colorectal cancer immunotherapy", 'colorectal cancer immunotherapy treatment', 'colon cancer immunotherapy side effects' and 'bowel cancer immunotherapy outcomes'.The incognito window on Google Chrome was used to conduct the search, and the browser was cleaned to reduce the effect of Google search on personalised search algorithms. 25learance of browser history, cache and cookies was done to make sure that previous searches do not affect our search results. 26Google search engine was selected as it is the absolute traffic leader and commonly used engine by the public. 18Yahoo and Bing web browsers were screened after searching Google.The search conducted on Google identifies most of the links incorporated using Yahoo and Bing which are also commonly used browsers by internet users.This finding was also discovered by Mac et al. 26 in a study done in Australia.
Previous research indicates that 90% of internet users view the first or second result pages (first 20 sites). 27So, the first 20 relevant (English) results were selected for each search term independently (n = 80).Duplicates (repeated appearance) were removed and noted for each search term and then across all search terms (n = 43).Websites were excluded (n = 20) if they are journals for healthcare providers (articles and books) (n = 9), commercial (biopharmaceutical companies) (n = 6), scientific research (n = 3) and non-relevant (other types of cancer or one paragraph provided) (n = 2).Websites were included if they provide information on immunotherapy for patients and their carers (n = 17) (Figure 1).

Data extraction
Data was extracted for the included websites on an Excel sheet by one author, for the following variables: website's country and affiliation category, scope of immunotherapy information, types of immunotherapy treatment and all the JAMA and DISCERN variables.Noteworthy, 80% of the public are looking for a quality seal on websites displaying medical information. 28Therefore, data were also extracted on whether the Health On the Net Foundation (HON) code (HONcode), Patient Information Forum (PIF) TICK and Better Business Bureau (BBB) seals were displayed on the website page. 26,28,29The process of data extraction was done in 5 consecutive days, starting from 20 July 2021 to 24 July 2021.
Two investigators independently evaluated the quality of websites using the JAMA and DISCERN instruments; discrepancies were discussed until we reach a consensus.As JAMA has very broad definition for its criteria, the two investigators agree on the same definitions for the presence of each JAMA criteria.Variation was found between the investigator's DISCERN score, so the average (mean) total score for each website was taken. 27

Data analysis
Descriptive narrative univariate analysis was done for all the variables including as follows: website's country and affiliation category, the scope of immunotherapy information and types of treatment discussed and all the JAMA benchmark and DISCERN variables.Narrative bivariate analysis was done to describe the relationship between the JAMA and DISCERN score and website's affiliation category.

JAMA benchmark
JAMA benchmark has four criteria to assess the quality of websites, including authorship, attribution, currency and disclosure.The presence of each criterion gives a score of one point. 26

DISCERN
Discern instrument is a critical appraisal tool with a validated questionnaire for evaluation of written user health information. 27It is composed of 3 sections as follows: Section 1 (questions 1-8) evaluates the reliability of the information, whereas Section 2 (questions 9-15) assesses the information quality about treatment choices, and Section 3 (question 16) reflects an overall scoring rate independently of the previous 15 questions.Each question from the 16 has a ranging scale from 1 to 5, 1 indicates definite no and 5 definite yes.Partial scoring of 2-4 suggests the presence of some elements of the question. 30Although the DICERN tool has a very clear clarification for its criteria, investigators agree on definite definition for yes and the opposite for no in all questions.Variation was noticed between investigator's partial scores.Total average results were rated from 'excellent' to 'very poor' as described in previous studies. 30

HONcode
The HONcode is a quality certification that reflects the website's adherence to the eight ethical principles, including authorship, complementarity, privacy, attribution, justifiability, transparency, financial disclosure and advertising policy.Websites were evaluated for the presence of a HONcode seal which is awarded to websites that comply with the abovementioned criteria. 31

PIF TICK
PIF TICK is the only UK quality mark for trustworthy health information.To be awarded a PIF TICK mark seal, the organisation must pass through strong and professional processing of health information and meet the ten required criteria 28 (Appendix C).

BBB accreditation
BBB is an organisation that reflects customers' reviews to create rating standards about the company.BBB accreditation is not applied for health information; it is awarded by a business that managed to make a good effort in resolving consumer complaints, and the seal displays the company's credibility. 32Table 4 in Appendix D presents the BBB standard criteria. 29

Results
Of the 17 websites included in our study, most of the websites are from the United States (n = 10, 58.8%), followed by the United Kingdom (n = 3, 17.6%), then Ireland (n = 2, 11.8%).Spain and Australia have the least presentation, each constituting n = 1 (5.9%) (Appendix E).This variation reflects ranking on Google search optimisation and English search.
Most included websites belong to charity affiliation category (n = 7, 41.2%), followed by academic and medical practice websites, each representing n = 3 (17.6%).Governmental and health portal have the least presentation, each constituting n = 2 (11.8%).US websites dominate its presence in most affiliation categories as follows: 100% of health portal, 66.6% of academic and 57.1% of charity websites (Appendix F).
Most of websites provide information on colorectal cancer immunotherapy (n = 13, 76.5%), while n = 4 (23.5%)give general cancer immunotherapy information, as presented in Table 7 in Appendix G. Websites presenting colorectal cancer information on immune checkpoint inhibitors constitute the majority (n = 9, 52.9%), indicating its recent approval by the Food and Drug Administration for treating microsatellite instability-high colorectal cancer. 3ewer websites were noticed to present colorectal cancer immunotherapy information on cancer vaccines, as they are in clinical testing. 3From the overall colorectal cancer information on immune checkpoint inhibitors (n = 9, 52.9%), n = 2 (22.2%) is presented as press news, while n = 1 (11.1%) is presented as an interesting blog created by oncology consultant colorectal cancer patient.
As shown in Most of the websites with a score of ≥3 points on JAMA or excellent/good on DISCERN belong to charity affiliation, n = 4 for each (Table 2).Moreover, there is an overlap of three charity websites meeting the abovementioned criteria; thus, the charity affiliation category dominates its presence on websites fulfilling ≥3 points and excellent/good on JAMA and DISCERN, respectively (Table 3).All governmental and medical practice and most of the academic websites have a score of <3 points on JAMA, while all governmental and academic and most medical practice have fair/poor DISCERN scores (Table 2).All websites with a governmental affiliation category score <3 points and fair/poor on JAMA and DISCERN, respectively (Table 3).
As shown in Table 1, all websites have full disclosure of ownership, sponsorship, advertising policy and sources of funding.A total of 58.8% of the websites present the date of copyright and/or the review date and provide up-to-date information.Overall, 41.2% of the websites have an author, while 29.4% wrote the sources of the information.Although one point is given for the presence of each criterion according to investigator's definitions, there is variation between websites in providing authorship, attribution of references and date of the information.
Table 3 presents all websites in descending order, with the highest quality score on DISCERN at the top.Total average DISCERN score ranges between 67 (highest) and 31(lowest).The top two websites according to the DISCERN evaluation meet all the JAMA criteria and have an additional accreditation presented as a PIF TICK or HONcode seal.Both provide information on colorectal cancer immunotherapy, and one is focusing on immune checkpoint inhibitors.The highest scoring is given to UK charity website with a PIF TICK mark seal.One HONcode seal is noted on included websites (5.8%).
As presented in Table 3, although good-quality websites show variation in total JAMA scoring between websites, the highest-scoring one on DISCERN meets all JAMA criteria.Most of the good-quality websites have a charity affiliation, and half of them give information on colorectal cancer immunotherapy.Fair-quality websites are dominant, with the highest scoring given to two US charity organisations with BBB accreditation, as shown in Table 3.All of them provide information on colorectal cancer immunotherapy, with the majority focusing on immune checkpoint inhibitors.All BBB-accredited charity websites are from the United States and have good/poor DISCERN score.
Poor-quality websites on DISCERN have lower scoring on JAMA (<3) with no additional credentials, as presented in Table 3.They show variation in country and affiliation presentation.They are the only category to provide detailed information on cancer vaccine.
As in Table 3, a notable feature of higher-quality websites is their comprehensive presentation of authorship and date currency.They present both an author and a reviewer, with the website's ranked no. 1 information written, reviewed and approved by the website team, which includes healthcare professionals and professors in oncology.The no. 2 ranked website's information is written and reviewed by two different individuals with the reviewer credentials given as a registered nurse.Both websites provide a last review date and date on sources.However, no.1 is distinguished by writing the next review date while no. 2 the date of copyright.
The following results are presented in Table 4. On applying the DISCERN questionnaire to the websites, gaps were noticed in providing the following: the prognosis of patients if no treatment is given (n = 0, 0%), clear aim (n = 4, 23.5%) and the effect of treatments on patient's quality of life (n = 7, 41.2%).All websites succeed to provide relevant information and describe the mechanism of action and benefits of treatments (n = 17, 100%).Most of the websites provide balanced information (n = 16, 94.1%), refer to areas of knowledge uncertainty (n = 15, 88.2%) and rated high to moderate on question 16 (n = 13, 76.4%).Small variation was noticed between websites in describing the following: more than one treatment choice (n = 12, 70.5%), support shared decision-making (n = 11, 64.7%), treatment risks (n = 11, 64.6%), details of additional sources of information (n = 11, 64.7%) and date on the information used or reported in the website (n = 10, 58.8%).Mohamed et al.Is it clear when the information used or reported in the publication was produced?Dates of the main sources of information used to compile the publication with the revision dates (not reprinting date) and the copyright date are provided.

(
6 Is it balanced and unbiased? The publication is written from an objective point of view, or a range of sources of information was used to compile the publication or evidence of an external assessment of the publication.As presented in Table 5 in Appendix H, the website ranked no. 3 on DISCERN shows the earliest presentation on Google search in all search terms, followed by no. 2 ranked website.The website ranked no. 1 appears on the first two search terms only and shows its latest presentation among others.The first search term 'colon cancer immunotherapy' has the highest exclusion and inclusion rate with no duplicates (Table 1 in Appendix A).It also shows the earliest appearance of the top three websites (10th, 6th, and 3rd) on the first page (first 10 hits), although in inverse order (Table 5 in Appendix H).Moreover, all the top 3 websites provide information specifically on colorectal cancer immunotherapy (Table 4).So, the first search term can be recommended as the best for patients seeking information on colorectal cancer immunotherapy.
Figure 2 in Appendix J presents a mind map for problem definition in our investigation.

Discussion
We found the majority of websites describing colorectal cancer immunotherapy were of low to moderate quality using the JAMA and fair/poor on DISCERN quality tools.A substantial variation in the information provided to patients on treatment effectiveness and impact of treatment on quality of life, authorship details and dating of information was noted.Additional accreditations such as a PIF TICK and HONcode seal were included in two websites.This study identifies gaps in online treatment information and missing details that denote a high quality of information.
A major finding of our study was that most of the websites scored <3 points on JAMA and fair/poor on DISCERN.This finding was also reported in studies done by Kaicker and Dang and Olkun and Demirkaya (2016,  2018). 27,30However, this doesn't support the previous research done in the United States where Google returns higher reliability scores than Yahoo and Bing and surprisingly MedlinePlus. 18Janssen et al. (2019) 33 study also confirms good-quality information about prostate cancer radiotherapy on the internet.
In our investigation, most of charity websites score ≥3 points and excellent/good on JAMA and DISCERN, respectively, indicating their higher reliability, in accordance with a study done by Janssen et al. (2019). 33ll governmental and most academic and medical practice websites were ranked fair/poor on DISCERN.This result is supported by Janssen et al.'s 33 (2019) study and disagrees with Corcelles et al.'s 34 (2015) research findings, which reported the commercial websites as the worst quality and the academic as the best.
Gaps were noticed in providing authorship, sources and date on website's information.These results match those observed in earlier studies done in Turkey and Canada (2018, 2016). 27,30st of the websites fail to mention clear aim; however, this has not previously been described in studies done in India and Canada (2015, 2016), where websites scored well in the categories of providing appropriate aim. 27,31ailure to describe the effects of treatments on patient's prognosis and quality of life was noticed in our investigation.These patterns were also seen in previous studies done by Kiran et al. and Kaicker and Dang (2015,  2016). 27,31he top two excellent websites on DISCERN meet all the JAMA criteria and have an additional accreditation.This observation was noted in a study done in Canada (2010) 35 which reported higher DISCERN score in websites with seals of approval and UK (2015) study where HONcode seals are present in websites with high JAMA score. 18This outcome is contrary to that of Meric et al. 36 and Laversin et al. 37 who found none of the sites with a seal complied with all four JAMA benchmark criteria and 70% of websites with a seal did not meet the standards for certification, respectively.
Few HONcode seals were noted in our study and reported in studies done by Borgmann et al. 38 on prostate cancer and radical prostatectomy, Alkhateeb's 39 group in prostate cancer and surgery study, 39 and Janssen et al. 33 in a study of prostate cancer and radiotherapy.These findings indicate lower HONcode certification frequency when the search is more focused.
Websites of excellent quality on DISCERN instrument were not top ranked by Google in any of the search terms used.This finding is consistent with a study done in Canada which reported higher-quality websites as not appearing on higher hits 27 and disagrees with a study done by Tan et al. 40 in a review of mesothelioma information on world wide web.

Implications for practice
Based on our findings, attributing website's information to authors whose background is clearly described and adding date of information would be simple steps to quickly improve the quality of information about cancer treatments online.Writing an aim with the scope and target audiences and the effect of treatments on prognosis and quality of life will dramatically improve audience convenience and accountability.
The infodemiology approach in this study is useful to describe the deficits with online information.However, there are numerous opportunities to improve the usefulness and quality of online information.Using the incognito window with clearance of browser history can be helpful for internet consumers to reduce the effect of personalised search algorithms.Increasing public awareness of the pitfalls with online information and how to critically analyse online information is necessary.Notwithstanding, seeking the views of people using the websites and collaborating with them can be a useful step in future research to improve reliability of online information.Moreover, the recommendation of high-quality resources for patients by healthcare providers can help to improve timely self-empowerment of patients with the accurate cancer information.

Strength and limitations
This investigation is the first to evaluate the quality of colorectal cancer immunotherapy information on websites using the DISCERN and JAMA tools combined with design thinking technique.We selected an unfamiliar treatment for most patients and oncology providers. 23The use of various quality stamps like HONcode and PIF TICK in the study is a strength.There are some limitations with this study.Only English websites are included.The internet is a dynamic process, and Google, as the only search engine used, is constantly updating its website ranking algorithm. 33t was not possible to include all patient websites for colorectal cancer, so high-quality websites may not be included in the assessment if they are not ranked in the top 20 by Google.There is an overlap between some of the affiliation categories used for websites, although they are done based on previous investigations. 18,25The limitation of using JAMA in our investigation is that simply perfect scoring is obtained by the presence of each criterion.As such, it is better used to measure the transparency of the websites as described in a study done by Silberg et al. 41 that first described the JAMA score.

Conclusion
Most of the websites discussing colorectal cancer immunotherapy are of low to moderate quality, so patients should be cautious about trusting the information provided on websites.Digital literacy training would be beneficial for improving lay audience identification of trustworthy information.The websites operated by charity organisations were of superior quality compared to hospital and sponsored medical news sites.Thus, non-commercial sites should be preferentially recommended to patients seeking online information.
Acknowledgements: Foremost, the authors would like to express their sincere gratitude to their supervisor Dervla Kelly for the continuous support of their master research, for her patience, motivation, enthusiasm and immense knowledge.Besides their supervisor, the authors would like to thank Prof. Desmond Leddin for giving his insightful comments.Their sincere thanks go to all the Master of Public Health team for the supporting research sessions.Their deepest gratitude goes to their husband, kids and all their family.Contributorship: H.M.: Conceptualisation, data curation, formal analysis, investigation, methodology, project administration, resources, validation, visualisation, and writingoriginal draft preparation.L.O.: Investigation (secondary review of website's evaluation).D.K.: Supervision and writingreview and editing.

Declaration of Conflicting Interests:
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Ethical approval: No ethical approval was required to conduct this study.

Funding:
The authors received no financial support for the research, authorship, and/or publication of this article.
Informed consent: Patient consent was not necessary, as the study was done on the literature and does not include human subjects.

Figure 1 .
Figure 1.Consort flow diagram of website's inclusion and exclusion for evaluation.

= 4 , 1 <3 10 ( 58 . 8 )= 2 , 1 27
Number of websites n (%) (Total n = 17) Number of websites by affiliation category for each JAMA score n (Total n = 17medical practice n = 2, academic n = Governmental n = 2, academic n = 2, charity n = 3, medical practice n = 3 DISCERN score (range = 16-80) 30 Number of websites n (%) (Total n = 17) Number of websites by affiliation category for each DISCERN score category (Total n = 17) academic n = 2, governmental n = 1, health portal n = 1, medical practice n = what sources of information were used to compile the publication (other than the author or producer)?The main statements made about treatment choices are accompanied by a reference to the sources associated with means of checking the sources used, such as a bibliography or external links to the online sources.

Table 1 .
JAMA criteria description, investigator's definition and number of websites adhering to investigator's definition with their specification.

Table 2 .
Categorisation of websites according to JAMA score, mean DISCERN score and website's affiliation category.

Table 3 .
For the 17 websites, information on country, affiliation category, information presentation, total JAMA benchmark score, mean DISCERN score and presence/absence of other accreditations in descending DISCERN score order is presented.

Table 4 .
DISCERN plus instrument (modified according to Borgmann et al. (2019)) questions, investigator's definition for full score and number of websites with yes, partial and no score.