Digital Scientific Platform for Independent Content in Neurology: Rigorous Quality Guideline Development and Implementation

Background Digital communication has emerged as a major source of scientific and medical information for health care professionals. There is a need to set up an effective and reliable methodology to assess and monitor the quality of content that is published on the internet. Objective The aim of this project was to develop content quality guidelines for Neurodiem, an independent scientific information platform dedicated to neurology for health care professionals and neuroscientists. These content quality guidelines are intended to be used by (1) content providers as a framework to meet content quality standards and (2) reviewers as a tool for analyzing and scoring quality of content. Methods Specific scientific criteria were designed using a 5-point scale to measure the quality of curated and original content published on the website: for Summaries, (1) source reliability and topic relevance for neurologists, (2) structure, and (3) scientific and didactic value; for Congress highlights, (1) relevance of congress selection, (2) congress coverage based on the original program, and (3) scientific and didactic value of individual abstracts; for Expert points of view and talks, (1) credibility (authorship) and topic relevance for neurologists, (2) scientific and didactic value, and (3) reliability (references) and format. The criteria were utilized on a monthly basis and endorsed by an independent scientific committee of widely recognized medical experts in neurology. Results Summary content quality for the 3 domains (reliability and relevance, structure, and scientific and didactic value) increased in the second month after the implementation of the guidelines. The domain scientific and didactic value had a mean score of 8.20/10. Scores for the domains reliability and relevance (8-9/10) and structure (45-55/60) showed that the maintenance of these 2 quality items over time was more challenging. Talks (either in the format of interviews or slide deck–supported scientific presentations) and expert point of view demonstrated high quality after the implementation of the content quality guidelines that was maintained over time (15-25/25). Conclusions Our findings support that content quality guidelines provide both (1) a reliable framework for generating independent high-quality content that addresses the educational needs of neurologists and (2) are an objective evaluation tool for improving and maintaining scientific quality level. The use of these criteria and this scoring system could serve as a standard and reference to build an editorial strategy and review process for any medical news or platforms.


Purpose
The objective of this document is to provide a scoring system to support Neurodiem content quality assessment. This document is intended to be distributed among internal and external stakeholders involved in reviewing and monitoring Neurodiem content towards smooth improvement and maintenance of translation and medical content quality as per BHS defined standards.

Scope
This quality measurement system applies to monitoring of criteria for translation and scientific quality for Neurodiem content as defined in section 3. of the document "Neurodiem content quality specifications".

Translation quality measurement
In order to perform monitoring of translation quality level and progression, qualification for 3 distinct groups of translation defaults, as per deviations to quality criteria depicted in section 3.1. of the document "Neurodiem content quality specifications" has been defined. These KPIs apply to any type of content stream subjected to translation, i.e. CSS, COH, IDS and Talks (videos) described in section 2.4. of the SOP "Neurodiem content quality control".
-Mild translation defaults: These include typos, spelling and grammar mistakes in local languages, when not resulting in any misunderstanding or misinterpretation, especially from a scientific point of view.
-Moderate translation defaults These include inadequate wording and improper use of scientific/medical glossary, but when these do not result in any misunderstanding or misinterpretation, especially from a scientific point of view.
-Severe translation defaults These include inadequate wording and improper use of scientific/medical glossary and when these result in any misunderstanding or misinterpretation, especially from a scientific point of view.

Scientific and medical quality measurement
In order to perform monitoring of scientific and medical quality level and progression, a quality measurement tool, based on a 5-point scale, has been set up to assess application of the quality criteria defined in section 3.2. of the document "Neurodiem content quality specifications". Instructions supporting use of this scoring system is presented below for each content stream.

CSS quality score
Neurodiem content scientific quality criteria and scoring system for article summaries (CSS) are described below. 3 main quality domains "Reliability and relevance", "Structure", "Scientific and didactic value" have been defined. Each of them includes quality items which are rated on a 5-point scale according to achievement of the described criteria. Scores in the individual sections are summed to generate a sub-score related to each quality domain and that is further weighted by a coefficient Confidential -For internal use only -Biogen Healthcare Solutions (indicated on the right upper hand side of the criteria grid). All sub-scores are summed to generate a total score of up to 100 corresponding to the overall quality of the CSS. Table 1 provided in the appendix corresponds to the template of the scoring table used by reviewers  when analyzing CSS scientific & medical quality. Reliability and relevance (source and topic) (score x1) Hindex in the subject category "Neurology (clinical)" (Scimago journal & country rank (SJR)) 1 -Article not curated from a peer-reviewed scientific journal 2 -Article curated from a journal that is not classified in the SJR filtered in the categories "Neurology", or "Neurology (clinical)", or "Neuroscience", or "Cognitive neuroscience" 3 -Article curated from a journal classified in the SJR in the category "Neurology (clinical)", with an H-index ranging <30 4 -Article curated from a journal classified in the SJR in the category "Neurology (clinical)", with an H-index ranging (30-69) 5 -Article curated from a journal classified in the SJR in the category "Neurology (clinical)", with an H-index ranging (70-351) Topic 1-Topic not related to the Neurology/Neuroscience field (off topic) 2-Topic in the neuroscience field, but not of specific interest for neurologists (poor contribution and impact in the field, or purely basic research issue with no interest to clinicians) 3-Topic being a relevant contribution in Neurology, but very far from any application in the clinical setting (e.g. advances in neurological disorders pathophysiologic mechanisms only) 4-Topic being a relevant and major contribution in the field, anticipated impact in the clinical practice as per author's conclusion 5-Topic being a relevant and major contribution in the field, with direct/immediate impact in the clinical practice as per author's conclusion 1-Does not reflect main findings of the source article 2-Scientifically accurate as per source content but not understandable for any neurologist sub-specialist audience 3-Scientifically accurate and understandable but not informative as per main findings of the source article 4-Scientifically accurate, understandable and informative as per source content but not attractive from a journalistic perspective 5-Scientifically accurate, understandable, informative and attractive.
Take away 1-Does not reflect main findings and conclusion of the source article (2-3) -Does partially reflect main findings of the source article and/or lack critical background information (4-5) -Does entirely reflect main findings of the source article and provides background information, towards being an independent section from the rest of the summary Why this matters? 1-Does not accurately describe clinical impact of the study results as per source article (2-3)-Lacks either scientific & medical novelties specifically brought by the study or potential impact of the study in the clinical setting (4-5) -Describe accurately and didactically scientific & medical novelties specifically brought by the study and potential impact of the study in the clinical setting Confidential -For internal use only -Biogen Healthcare Solutions

Study design
Considering the following items: study objective, endpoint(s), population/model characteristics, study design, study duration, procedures and investigations (description and rationale): are these elements present and accurately presented? 1-None of these items are represented or accurately described as per source content (2-3) Only some of these items are represented or may be not accurately described and explained (4-5) All of these items are represented and are accurately described and explained Key results 1-Results described are inaccurate as per original article (e.g. may provide wrong or biased scientific or clinical conclusion as per original article) (2-3)-Results described are scientifically accurate but partially reflect author's conclusion or size effect/clinical value of data are not enough illustrated (e.g. numerical data, p) 4-Results described scientifically accurate; clinical value well illustrated, but no editorial selection performed to highlight only major results as per author's conclusion 5-Results described scientifically accurate; clinical value well illustrated; selection among results performed to present main findings supporting author's conclusion.

Accuracy
Overall, does the article summary restitute accurately: scientific glossary/abbreviations, numerical and statistical data, as well as all methodology, results, interpretation, conclusion and clinical impact of the study? 1-Strongly disagree 2-Disagree 3-Neutral 4-Agree

Didactic dimension
Overall, does the article provide sufficient background information (study contextualization, scientific definition) toward being understandable by any neurologist profiles; is it written logically and coherently so that the reading

COH quality score
Neurodiem content scientific quality criteria and scoring system for "congress highlights" (COH) are described below. 3 main quality domains "Congress selection relevance", "Congress coverage", "Scientific and didactic value" have been defined.

Confidential -For internal use only -Biogen Healthcare Solutions
Each of them includes quality items which are rated on a 5-point scale according to achievement of the described criteria. Scores in the individual sections are summed to generate a sub-score related to each quality domain, which is further weighted by a coefficient (indicated on the right upper hand side of the criteria grid). All sub-scores are summed to generate a total score of up to 100 corresponding to the overall quality of the congress highlights encompassing relevance of congress selection and topic coverage as well as mean scientific and didactic value of a representative panel of abstracts analyzed at individual level.   Confidential -For internal use only -Biogen Healthcare Solutions

FAL quality score
No specific measurement scoring system has been developed for FALs which are licensed content that already went through a peer-review process and are selected among a defined set of journals in Neuroscience/Neurology. BHS quality review stakeholders will however address whether the following quality criteria are met based on the analysis of the FAL set delivered on a monthly basis: -Currency of articles: Neurodiem publication date within one month from the original publication date; -Balance between the 18 Neurodiem topics (sub-specialties/areas of interest).

IDS and Talks quality score
Neurodiem content scientific quality criteria and scoring system for original content, i.e. IDS and Talks are described below. 3 main quality domains "Credibility and relevance", "Scientific value and didactic dimension", "Reliability and format" have been defined. Each of them includes quality items which are rated on a 5-point scale according to achievement of the described criteria. Scores in the individual sections are summed to generate a sub-score related to each quality domain which is further weighted by a coefficient (indicated on the right upper hand side of the criteria grid). All sub-scores are summed to generate a total score of up to 100 corresponding to the overall quality of the IDS or Talks (scientific presentation or interview).

Authors/speakers
Does the author/speaker meet quality standards in terms of academic seniority, reputation among peers in Neurology/Neuroscience, experience and recognized publications in the topic/field addressed in the IDS/Talks? 1-Strongly disagree 2-Disagree 3-Neutral 4-Agree

5-Strongly agree
Confidential -For internal use only -Biogen Healthcare Solutions Topic 1-Topic not related to the Neurology/Neuroscience field (off topic) 2-Topic in the neuroscience field, but not of specific interest for neurologists (poor contribution and impact in the field, or purely basic research issue with no interest to clinicians) 3-Topic being a relevant contribution in Neurology, but very far from any application in the clinical setting (e.g. advances in neurological disorders pathophysiologic mechanisms only) 4-Topic being a relevant and major contribution in the field, anticipated impact in the clinical practice as per author's conclusion 5-Topic being a relevant and major contribution in the field, with direct/immediate impact in the clinical practice as per author's conclusion 3-Neutral (7-9 min) 4-Agree (5-7 min)

Total score expert points of view (IDS)/Talks (overall quality): /100
Neurodiem content quality measurement -v.   Total COH quality score (for one congress) up to a score of 100 = sub-score 1 (/5) + sub-score 2 (/30) + mean sub-score 3 (/65) for a representative panel of abstracts (n>=8) analyzed at individual level Confidential -For internal use only -Biogen Healthcare Solutions