Examining Care Planning Efficiency and Clinical Decision Support Adoption in a System Tailoring to Nurses’ Graph Literacy: National, Web-Based Randomized Controlled Trial

Background The proliferation of health care data in electronic health records (EHRs) is fueling the need for clinical decision support (CDS) that ensures accuracy and reduces cognitive processing and documentation burden. The CDS format can play a key role in achieving the desired outcomes. Building on our laboratory-based pilot study with 60 registered nurses (RNs) from 1 Midwest US metropolitan area indicating the importance of graph literacy (GL), we conducted a fully powered, innovative, national, and web-based randomized controlled trial with 203 RNs. Objective This study aimed to compare care planning time (CPT) and the adoption of evidence-based CDS recommendations by RNs randomly assigned to 1 of 4 CDS format groups: text only (TO), text+table (TT), text+graph (TG), and tailored (based on the RN’s GL score). We hypothesized that the tailored CDS group will have faster CPT (primary) and higher adoption rates (secondary) than the 3 nontailored CDS groups. Methods Eligible RNs employed in an adult hospital unit within the past 2 years were recruited randomly from 10 State Board of Nursing lists representing the 5 regions of the United States (Northeast, Southeast, Midwest, Southwest, and West) to participate in a randomized controlled trial. RNs were randomly assigned to 1 of 4 CDS format groups—TO, TT, TG, and tailored (based on the RN’s GL score)—and interacted with the intervention on their PCs. Regression analysis was performed to estimate the effect of tailoring and the association between CPT and RN characteristics. Results The differences between the tailored (n=46) and nontailored (TO, n=55; TT, n=54; and TG, n=48) CDS groups were not significant for either the CPT or the CDS adoption rate. RNs with low GL had longer CPT interacting with the TG CDS format than the TO CDS format (P=.01). The CPT in the TG CDS format was associated with age (P=.02), GL (P=.02), and comfort with EHRs (P=.047). Comfort with EHRs was also associated with CPT in the TT CDS format (P<.001). Conclusions Although tailoring based on GL did not improve CPT or adoption, the study reinforced previous pilot findings that low GL is associated with longer CPT when graphs were included in care planning CDS. Higher GL, younger age, and comfort with EHRs were associated with shorter CPT. These findings are robust based on our new innovative testing strategy in which a diverse national sample of RN participants (randomly derived from 10 State Board of Nursing lists) interacted on the web with the intervention on their PCs. Future studies applying our innovative methodology are recommended to cost-effectively enhance the understanding of how the RN’s GL, combined with additional factors, can inform the development of efficient CDS for care planning and other EHR components before use in practice.

TITLE AND ABSTRACT 1a) TITLE: Identification as a randomized trial in the title 1a) Does your paper address CONSORT item 1a? * I.e does the title contain the phrase "Randomized Controlled Trial"? (if not, explain the reason under "other")

Yes
Other: 1a-i) Identify the mode of delivery in the title Identify the mode of delivery. Preferably use "web-based" and/or "mobile" and/or "electronic game" in the title. Avoid ambiguous terms like "online", "virtual", "interactive". Use "Internet-based" only if Intervention includes non-web-based Internet components (e.g. email), use "computer-based" or "electronic" only if offline products are used. Use "virtual" only in the context of "virtual reality" (3-D worlds). Use "online" only in the context of "online support groups". Complement or substitute product names with broader terms for the class of products (such as "mobile" or "smart phone" instead of "iphone"), especially if the application runs on different platforms.
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 1a-i? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer "web based" 1a-ii) Non-web-based components or important co-interventions in title Mention non-web-based components or important co-interventions in title, if any (e.g., "with telephone support").

Your answer
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 1a-ii? Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 1a-iii) Primary condition or target group in the title Mention primary condition or target group in the title, if any (e.g., "for children with Type I Diabetes") Example: A Web-based and Mobile Intervention with Telephone Support for Children with Type I Diabetes: Randomized Controlled Trial subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 1a-iii? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 1b) ABSTRACT: Structured summary of trial design, methods, results, and conclusions NPT extension: Description of experimental treatment, comparator, care providers, centers, and blinding status.
1b-i) Key features/functionalities/components of the intervention and comparator in the METHODS section of the ABSTRACT Mention key features/functionalities/components of the intervention and comparator in the abstract. If possible, also mention theories and principles used for designing the site. Keep in mind the needs of systematic reviewers and indexers by including important synonyms. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) subitem not at all important essential 1 2 3 4 5 There was no non-web-based component.
"nurse" Does your paper address subitem 1b-i? * Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/ relevant for your study Your answer 1b-ii) Level of human involvement in the METHODS section of the ABSTRACT Clarify the level of human involvement in the abstract, e.g., use phrases like "fully automated" vs. "therapist/nurse/care provider/physician-assisted" (mention number and expertise of providers involved, if any). (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 1b-ii? Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/ relevant for your study Your answer 1b-iii) Open vs. closed, web-based (self-assessment) vs. face-to-face assessments in the METHODS section of the ABSTRACT Mention how participants were recruited (online vs. offline), e.g., from an open access website or from a clinic or a closed online user group (closed usergroup trial), and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment). Clearly say if outcomes were self-assessed through questionnaires (as common in webbased trials). Note: In traditional offline trials, an open trial (open-label trial) is a type of clinical trial in which both the researchers and participants know which treatment is being administered. To avoid confusion, use "blinded" or "unblinded" to indicated the level of blinding instead of "open", as "open" in web-based trials usually refers to "open access" (i.e. participants can self-enrol). (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 1b-iii? Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/ relevant for your study Your answer "four CDS format groups; (1) text only; 2) text+table; 3) text+graph; 4) tailored (based on RN's graph literacy score)" The intervention was the tailoring of CDS format to RN graph literacy, which was automated.
"Eligible RNs employed in an adult hospital unit within the past 2 years were recruited randomly from 10 state nursing licensure lists representing the five regions of the United States" This was a pure web-based trial. Participants were blinded.
1b-iv) RESULTS section in abstract must contain use data Report number of participants enrolled/assessed in each group, the use/uptake of the intervention (e.g., attrition/adherence metrics, use over time, number of logins etc.), in addition to primary/secondary outcomes. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 1b-iv? Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/ relevant for your study Your answer 1b-v) CONCLUSIONS/DISCUSSION in abstract for negative trials Conclusions/Discussions in abstract for negative trials: Discuss the primary outcome -if the trial is negative (primary outcome not changed), and the intervention was not used, discuss whether negative results are attributable to lack of uptake and discuss reasons. (Note: Only report in the abstract what the main paper is reporting. If this information is subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 1b-v? Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/ relevant for your study Your answer INTRODUCTION 2a) In INTRODUCTION: Scientific background and explanation of rationale 2a-i) Problem and the type of system/solution Describe the problem and the type of system/solution that is object of the study: intended as stand-alone intervention vs. incorporated in broader health care program? Intended for a particular patient population? Goals of the intervention, e.g., being more cost-effective to other interventions, replace or complement other solutions? (Note: Details about the intervention are provided in "Methods" under 5) subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 2a-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer "The differences between the tailored (n=46) and the non-tailored (TO [n=55], TT [n=54], TG [n=48]) CDS groups were not significant for either the CPT or the CDS adoption rate." Negative outcome could not be attributed to lack of uptake.
"With the proliferation of health care data, there is an overwhelming amount of information available to support clinical decision-making if appropriately analyzed and delivered to the clinician in a format that is genuinely helpful. In spite of the length of time electronic health records (EHRs) have been around, there continues to be multiple reports of the unintended consequences occurring from poor design and inadequate testing. [1-5] These consequences negatively impact patients and clinicians. Nurses report major frustrations interacting with overly complicated EHR user interfaces as they seek to locate, understand, and apply pertinent information to patients' care.
[1] 2a-ii) Scientific background, rationale: What is known about the (type of) system Scientific background, rationale: What is known about the (type of) system that is the object of the study (be sure to discuss the use of similar systems for other conditions/diagnoses, if appropiate), motivation for the study, i.e. what are the reasons for and what is the context for this specific study, from which stakeholder viewpoint is the study performed, potential impact of findings [2]. Briefly justify the choice of the comparator.
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 2a-ii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 2b) In INTRODUCTION: Specific objectives or hypotheses Does your paper address CONSORT subitem 2b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer METHODS 3a) Description of trial design (such as parallel, factorial) including allocation ratio Does your paper address CONSORT subitem 3a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer "One way to enhance EHRs is to develop clinical decision support (CDS) systematically in steps that include ample testing prior to implementation in real-time. CDS is here defined as computer generated patient information and clinical knowledge that is filtered and presented when needed for decision making.
[7] Testing in real-world practice can improve the usability and effectiveness but has the potential for unintended patient harm. In contrast, conducting simulation-based trials prior to real-world testing provides a do-no-harm approach that identifies the potential for negative patient outcomes [8][9][10][11] before use with real patients. [12][13][14][15] A systematic review of 28 CDS studies targeting hospital nurses' decision-making shows the promise of this approach including: improved care processes in 7/7 studies, improved usability in 4/4 studies, and improved patient outcomes in 3/4 studies. [16] There were, however, important weaknesses in this body of research. Only 1 study employed a RCT design. Only two studies attempted to investigate the important issues of CDS format design, one using focus groups [17] and the other using observational data to compare the effect of a graphical display against a tabular display of patient data in ICUs. [18] Since that review, a 2017 study identified that bar graphs facilitated nurses' understanding over 3 other display formats, but comprehension was not optimal across formats (41-88% correct).
[19] " "our primary aim was to examine the effects of the tailoring (matching the RN's graphic literacy to the format) by comparing four randomly assigned CDS format groups, text, text+table, text+graph, and tailored, on RN care planning time and adoption of evidence based best practices. We hypothesized that the tailored CDS group would have faster care planning time (primary) and more favorable CDS recommendation acceptance rates." "This study was a national web based randomized control trial (RCT) with registered nurse (RN) subjects randomized to one of four CDS format groups: text only (TO), text+table (TT), text+graph (TG), or tailored. RNs assigned to the first three groups interacted with the corresponding CDS format condition, while RNs assigned to the tailored CDS group interacted with the CDS format 3b) Important changes to methods after trial commencement (such as eligibility criteria), with reasons Does your paper address CONSORT subitem 3b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 3b-i) Bug fixes, Downtimes, Content Changes Bug fixes, Downtimes, Content Changes: ehealth systems are often dynamic systems. A description of changes to methods therefore also includes important changes made on the intervention or comparator during the trial (e.g., major bug fixes or changes in the functionality or content) (5-iii) and other "unexpected events" that may have influenced study design such as staff changes, system failures/downtimes, etc. [2].
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 3b-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 4a) Eligibility criteria for participants Does your paper address CONSORT subitem 4a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer condition matched to their graph literacy scores, texts only for those with low score, text+table for those with medium score, and text+graph for those with high score. The definition of the score cutoff can be found below in the study measure section. It was approved by the Universities of Florida (IRB201902611) and Iowa (201910213) Institutional Review Boards. For additional details of our method see our study protocol published elsewhere.
[28] " There was no substantial change after the trial started.
In the beginning of the trial, a programming error caused all subjects to be assigned to non-tailored group. The bug was identified and fixed quickly.
"Nurses were eligible if they: 1) were 18 years or older, 2) possessed an RN license, 3) worked on an adult medical-surgical unit in a hospital setting within the last 2 years. The exclusion criteria were: 1) had limited or no access to the internet and/or a desktop, laptop, or tablet with a screen larger than 9 inches; 2) unable to complete the study due to physical or cognitive impairment or 3) unable to speak, read, or write English. To further enhance generalizability, quotas were set to ensure representation of important subgroups; males (goal 20%), minority nurses (goal 35%) and nurses with ADN 4a-i) Computer / Internet literacy Computer / Internet literacy is often an implicit "de facto" eligibility criterion -this should be explicitly clarified.
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 4a-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 4a-ii) Open vs. closed, web-based vs. face-to-face assessments: Open vs. closed, web-based vs. face-to-face assessments: Mention how participants were recruited (online vs. offline), e.g., from an open access website or from a clinic, and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment), i.e., to what degree got the study team to know the participant. In online-only trials, clarify if participants were quasi-anonymous and whether having multiple identities was possible or whether technical or logistical measures (e.g., cookies, email confirmation, phone calls) were used to detect/prevent these.
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 4a-ii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer as their highest degree (goal 30%). We aimed to recruit about 20% from each of the five US Regions. " Our eligibility required internet access and cognitive ability to complete the web-based study.
Participants were recruited by mail, email or mix of both methods. Interactions with participants during recruitment were carried out by phone call, email and/or text. Communication occurred during the testing phase (session 2) via zoom meeting (camera was not required though).
4a-iii) Information giving during recruitment Information given during recruitment. Specify how participants were briefed for recruitment and in the informed consent procedures (e.g., publish the informed consent documentation as appendix, see also item X26), as this information may have an effect on user self-selection, user expectation and may also bias results.
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 4a-iii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 4b) Settings and locations where the data were collected Does your paper address CONSORT subitem 4b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 4b-i) Report if outcomes were (self-)assessed through online questionnaires Clearly report if outcomes were (self-)assessed through online questionnaires (as common in web-based trials) or otherwise.
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 4b-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
Data collection occurred in two sessions. Session 1: Screening and consent. This session involved screening, consenting (using an online process), collecting subject demographics and assessment of the RN subject's graph literacy score. Data were collected and securely stored in REDCap database hosted by the clinical and Translational Science Institute at the University of Florida (UF). After completing the graph literacy assessment in Session 1, each pre-qualified RN (met eligibility and quota considerations) was scheduled for Session 2 (CDS intervention).
Session 2: Experimental CDS intervention. In this session, data related to the experimental intervention and the post survey were collected. The amount of time needed to complete both sessions was approximately 1-2 hours. The data collected in this session were securely stored in SQL database server at UF Health.
In session 1, subjects were given 2 surveys 1) demographic and experience survey, and 2) a graph literacy survey they completed while through the REDCap system via a URL link sent to their email. In session 2, the participants complete a 35 question survey on usability, ease of use, experience with the intervention, and numeracy.
4b-ii) Report how institutional affiliations are displayed Report how institutional affiliations are displayed to potential participants [on ehealth media], as affiliations with prestigious hospitals or universities may affect volunteer rates, use, and reactions with regards to an intervention.(Not a required item -describe only if this may bias results) subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 4b-ii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
5) The interventions for each group with sufficient details to allow replication, including how and when they were actually administered 5-i) Mention names, credential, affiliations of the developers, sponsors, and owners Mention names, credential, affiliations of the developers, sponsors, and owners [6] (if authors/ evaluators are owners or developer of the software, this needs to be declared in a "Conflict of interest" section or mentioned elsewhere in the manuscript).
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 5-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 5-ii) Describe the history/development process Describe the history/development process of the application and previous formative evaluations (e.g., focus groups, usability testing), as these will have an impact on adoption/use rates and help with interpreting results.
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 5-ii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

5-iii) Revisions and updating
Revisions and updating. Clearly mention the date and/or version number of the application/ intervention (and comparator, if applicable) evaluated, or describe whether the intervention underwent major changes during the evaluation process, or whether the development and/ or content was "frozen" during the trial. Describe dynamic components such as news feeds or changing content which may have an impact on the replicability of the intervention (for unexpected events see item 3b).
Institutional affiliation was only displayed on recruitment material and informed consent.
The software was developed by our team in house.
"The 1st year of the R01 study was spent reprogramming and iteratively testing the new on-line version of the intervention. Reprogramming was performed by our team programmer and initial testing was conducted by the team. In the final phase of development, the entire procedure for accessing, orienting, and interacting with the intervention was pretested with a sample 31 RNs (10 from our initial study recruitment cycles and 21 from a convenience sample)." subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 5-iii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 5-iv) Quality assurance methods Provide information on quality assurance methods to ensure accuracy and quality of information provided [1], if applicable.
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 5-iv? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 5-v) Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing flowcharts of the algorithms used Ensure replicability by publishing the source code, and/or providing screenshots/screencapture video, and/or providing flowcharts of the algorithms used. Replicability (i.e., other researchers should in principle be able to replicate the study) is a hallmark of scientific reporting.
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 5-v? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
There was no major change during the trial.
"Zoom videoconferencing software (Zoom Video Communications, San Jose, CA) was integrated into the intervention so study staff could orient RN subjects before RN interaction with the application and remain present in the background to identify and intervene if technology issues occurred during RN interaction with the application." We provided screenshots of CDS formats. The tailoring method is described below: "In our study, the GLS score was used to assign tailored CDS group RNs to format conditions as follows: RNs with graph literacy scores up to 10 were presented with the text format; those with graph literacy score of 11 were presented with the text+table format; and the RN with graph literacy scores 12 or higher were presented with the text+graph format.
[28] This tailoring method was motivated by our pilot findings. [27]" 5-vi) Digital preservation Digital preservation: Provide the URL of the application, but as the intervention is likely to change or disappear over the course of the years; also make sure the intervention is archived (Internet Archive, webcitation.org, and/or publishing the source code or screenshots/videos alongside the article). As pages behind login screens cannot be archived, consider creating demo pages which are accessible without login..
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 5-vi? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study 5-vii) Access Access: Describe how participants accessed the application, in what setting/context, if they had to pay (or were paid) or not, whether they had to be a member of specific group. If known, describe how participants obtained "access to the platform and Internet" [1]. To ensure access for editors/reviewers/readers, consider to provide a "backdoor" login account or demo mode for reviewers/readers to explore the application (also important for archiving purposes, see vi).
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 5-vii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
We have included screenshots with the article.
Participants accessed the intervention by a unique URL generated by the server system via the chat feature of the zoom meeting software system during session 2. Once the participant completed the post survey, the access to the system is closed and cannot be reaccessed. 5-viii) Mode of delivery, features/functionalities/components of the intervention and comparator, and the theoretical framework Describe mode of delivery, features/functionalities/components of the intervention and comparator, and the theoretical framework [6] used to design them (instructional strategy [1], behaviour change techniques, persuasive features, etc., see e.g., [7,8] for terminology). This includes an in-depth description of the content (including where it is coming from and who developed it) [1]," whether [and how] it is tailored to individual circumstances and allows users to track their progress and receive feedback" [6]. This also includes a description of communication delivery channels and -if computer-mediated communication is a componentwhether communication was synchronous or asynchronous [6]. It also includes information on presentation strategies [1], including page design principles, average amount of text on pages, presence of hyperlinks to other resources, etc. [1].
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 5-viii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 5-ix) Describe use parameters Describe use parameters (e.g., intended "doses" and optimal timing for use). Clarify what instructions or recommendations were given to the user, e.g., regarding timing, frequency, heaviness of use, if any, or was the intervention used ad libitum.
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 5-ix? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 5-x) Clarify the level of human involvement Clarify the level of human involvement (care providers or health professionals, also technical assistance) in the e-intervention or as co-intervention (detail number and expertise of professionals involved, if any, as well as "type of assistance offered, the timing and frequency of the support, how it is initiated, and the medium by which the assistance is delivered". It may be necessary to distinguish between the level of human involvement required for the trial, and the level of human involvement required for a routine application outside of a RCT setting (discuss under item 21 -generalizability).
subitem not at all important essential 1 2 3 4 5 The main intervention is the tailoring of CDS format (text vs text+table vs text +graph) based on RN graph literacy. This national study was built on a pilot study that indicated graphic literacy may have an impact. Two publications provide in-depth context for this study in addition to the multiple figures and tables of key informaton: The tailoring occurs once when a user first interacts with a CDS system and takes a graph literacy assessment.
Does your paper address subitem 5-x? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 5-xi) Report any prompts/reminders used Report any prompts/reminders used: Clarify if there were prompts (letters, emails, phone calls, SMS) to use the application, what triggered them, frequency etc. It may be necessary to distinguish between the level of prompts/reminders required for the trial, and the level of prompts/reminders for a routine application outside of a RCT setting (discuss under item 21generalizability).
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 5-xi? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 5-xii) Describe any co-interventions (incl. training/support) Describe any co-interventions (incl. training/support): Clearly state any interventions that are provided in addition to the targeted eHealth intervention, as ehealth intervention may not be designed as stand-alone intervention. This includes training sessions and support [1]. It may be necessary to distinguish between the level of training required for the trial, and the level of training for a routine application outside of a RCT setting (discuss under item 21generalizability. subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 5-xii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
The tailoring is automated.
Not applicable as user only interacted with the application once.
There was no co-intervention. 6a) Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed Does your paper address CONSORT subitem 6a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 6a-i) Online questionnaires: describe if they were validated for online use and apply CHERRIES items to describe how the questionnaires were designed/deployed If outcomes were obtained through online questionnaires, describe if they were validated for online use and apply CHERRIES items to describe how the questionnaires were designed/deployed [9].
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 6a-i?

Copy and paste relevant sections from manuscript text
Your answer 6a-ii) Describe whether and how "use" (including intensity of use/dosage) was defined/measured/monitored Describe whether and how "use" (including intensity of use/dosage) was defined/measured/ monitored (logins, logfile analysis, etc.). Use/adoption metrics are important process outcomes that should be reported in any ehealth trial.
subitem not at all important essential 1 2 3 4 5 The primary outcome of this RCT was care planning time, defined as the total amount of time (in minutes) the RN spent interacting with the care planning software over the three simulated shifts. The secondary outcome was the CDS adoption rate, which was calculated based on the RN accepting CDS suggestions for improving the simulated patients' outcomes.
The main focus of our study was evaluating the interactions of nurses with software. All of those who participated in the study completed questions related to the user experience with the software and thus this was NOT an internet survey.
Does your paper address subitem 6a-ii? Copy and paste relevant sections from manuscript text Your answer 6a-iii) Describe whether, how, and when qualitative feedback from participants was obtained Describe whether, how, and when qualitative feedback from participants was obtained (e.g., through emails, feedback forms, interviews, focus groups).
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 6a-iii? Copy and paste relevant sections from manuscript text Your answer 6b) Any changes to trial outcomes after the trial commenced, with reasons Does your paper address CONSORT subitem 6b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 7a) How sample size was determined NPT: When applicable, details of whether and how the clustering by care provides or centers was addressed 7a-i) Describe whether and how expected attrition was taken into account when calculating the sample size Describe whether and how expected attrition was taken into account when calculating the sample size.
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 7a-i? Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
Not applicable as users interacted with the application only once.
We did not obtain qualitative feedback.
There was no change.
"Our aim was to estimate the effect of tailoring on RN care planning time. Therefore, we focused on the comparison between the tailored CDS group and each of the three non-tailored CDS groups, text, text+table, and text+graph. To achieve a family-wise type 1 error of 5%, we set the cutoff for significance at 0.016 using Bonferroni adjustment. Using the findings from our pilot study on the association between GL and RN care planning time, [27] we estimated 7b) When applicable, explanation of any interim analyses and stopping guidelines Does your paper address CONSORT subitem 7b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 8a) Method used to generate the random allocation sequence NPT: When applicable, how care providers were allocated to each trial group Does your paper address CONSORT subitem 8a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 8b) Type of randomisation; details of any restriction (such as blocking and block size) Does your paper address CONSORT subitem 8b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
We did not perform any interim analysis.
We used the psudonumber generator provided in the stat package of R.
Stratified by graph literacy (low, medium, high) block randomization with a block size of 8. 9) Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned Does your paper address CONSORT subitem 9? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
10) Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions Does your paper address CONSORT subitem 10? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 11a) If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how NPT: Whether or not administering co-interventions were blinded to group assignment 11a-i) Specify who was blinded, and who wasn't Specify who was blinded, and who wasn't. Usually, in web-based trials it is not possible to blind the participants [1, 3] (this should be clearly acknowledged), but it may be possible to blind outcome assessors, those doing data analysis or those administering cointerventions (if any).
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 11a-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
The randomization table was stored in SQL database and used by the software to decide the group assignment when a user access the application online.
The random allocation sequence was generated by the study statistician (Yingwei Yao) at University of Florida. Participant enrollment was conducted by Jessica Wolf at University of Iowa. Group assignment was performed by the software.
Both the participant and the research assistant providing orientation were blinded. This is possible since a user could be presented with a CDS format (text, text+table, text+graph) by being assigned to the corresponding non-tailored (control) group or being assigned to the tailored (intervention) group.
11a-ii) Discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator" Informed consent procedures (4a-ii) can create biases and certain expectations -discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator".
subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 11a-ii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
11b) If relevant, description of the similarity of interventions (this item is usually not relevant for ehealth trials as it refers to similarity of a placebo or sham intervention to a active medication/intervention) Does your paper address CONSORT subitem 11b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 12a) Statistical methods used to compare groups for primary and secondary outcomes NPT: When applicable, details of whether and how the clustering by care providers or centers was addressed Does your paper address CONSORT subitem 12a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 12a-i) Imputation techniques to deal with attrition / missing values Imputation techniques to deal with attrition / missing values: Not all participants will use the intervention/comparator as intended and attrition is typically high in ehealth trials. Specify how participants who did not use the application or dropped out from the trial were treated in the statistical analysis (a complete case analysis is strongly discouraged, and simple imputation techniques such as LOCF may also be problematic [4]).
subitem not at all important essential 1 2 3 4 5 The participant was blinded and hence did not know.
Tailoring was performed by the software behind the scene. Both control and intervention group can be presented with any of the three CDS formats; text only, text+table, and text+graph.
Regression analysis was used to estimate the intervention effect.
Does your paper address subitem 12a-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

12b) Methods for additional analyses, such as subgroup analyses and adjusted analyses
Does your paper address CONSORT subitem 12b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Does your paper address subitem X26-ii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
subitem not at all important essential 1 2 3 4 5 X26-iii) Safety and security procedures Safety and security procedures, incl. privacy considerations, and any steps taken to reduce the likelihood or detection of harm (e.g., education and training, availability of a hotline) Does your paper address subitem X26-iii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

RESULTS
13a) For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome NPT: The number of care providers or centers performing the intervention in each group and the number of patients treated by each care provider in each center Does your paper address CONSORT subitem 13a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
13b) For each group, losses and exclusions after randomisation, together with reasons Does your paper address CONSORT subitem 13b? (NOTE: Preferably, this is shown in a CONSORT flow diagram) * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Consent was obtained online after screening.
The user data was stored on secure REDCap and SQL database servers hosted by University of Florida to ensure data security and privacy.
"Of the 211 RNs (56 in TO, 55 in TT, 53 in TG, and 47 in tailored) in the main study, 8 (1 in TO, 1 in TT, 5 in TG, and 1 in tailored) were excluded from analysis due to significant interruption during the session for either real life circumstances or technical difficulties." "8 (1 in TO, 1 in TT, 5 in TG, and 1 in tailored) were excluded from analysis due to significant interruption during the session for either real life circumstances or technical difficulties." 13b-i) Attrition diagram Strongly recommended: An attrition diagram (e.g., proportion of participants still logging in or using the intervention/comparator in each group plotted over time, similar to a survival curve) or other figures or tables demonstrating usage/dose/engagement.
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 13b-i? Copy and paste relevant sections from the manuscript or cite the figure number if applicable (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
14a) Dates defining the periods of recruitment and follow-up Does your paper address CONSORT subitem 14a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
14a-i) Indicate if critical "secular events" fell into the study period Indicate if critical "secular events" fell into the study period, e.g., significant changes in Internet resources available or "changes in computer hardware or Internet delivery resources" subitem not at all important essential 1 2 3 4 5 Not applicable since this was a one time use intervention.
"The data reported for this intervention study were collected from RN subjects between June 2019 and January 2022 in two sessions (Figure 3 -Flow of national internet-based study)." Does your paper address subitem 14a-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
14b) Why the trial ended or was stopped (early) Does your paper address CONSORT subitem 14b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
15) A table showing baseline demographic and clinical characteristics for each group NPT: When applicable, a description of care providers (case volume, qualification, expertise, etc.) and centers (volume) in each group Does your paper address CONSORT subitem 15? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
There was no such event.
The trial ended at the end of the planned study period and after we have met enrollment target. See Table 2 in manuscript.
15-i) Report demographics associated with digital divide issues In ehealth trials it is particularly important to report demographics associated with digital divide issues, such as age, education, gender, social-economic status, computer/Internet/ehealth literacy of the participants, if known. subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 15-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
16) For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups 16-i) Report multiple "denominators" and provide definitions Report multiple "denominators" and provide definitions: Report N's (and effect sizes) "across a range of study participation [and use] thresholds" [1], e.g., N exposed, N consented, N used more than x times, N used more than y weeks, N participants "used" the intervention/comparator at specific pre-defined time points of interest (in absolute and relative numbers per group). Always clearly define "use" of the intervention.
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 16-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer "The RN completers aged 22-69 years with a mean age of 40.8 ± 11.7 years (median 39 years, 34% 18-34, 41% 35-49, 22% 50-64, 3% 65+). The subjects were comprised of 20% males, 75% Whites (66% Non-Hispanic White), 10% Blacks, 9% Asian, 5% other races, and 13% Hispanics. 26% of the participants had Associate Degrees in Nursing (ADN) or lower whereas 74% had Bachelor Degrees in Nursing (BSN) or higher. On average, the participants had 12.3 ± 10.5 years of RN experience, with 55% having up to 10 years, 27% 11-20, 9% 21-30, 7% 31-40, and 1% 40+ years' experience, and identified themselves as very comfortable (8.6 ± 1.5 out of 10) working with the EHR. The average GLS score among participants was 11.0 ± 1.6. " Text only group: 56 exposed, 55 included in analysis. Text+table group: 55 exposed, 54 included in analysis. Text+graph group: 53 exposed, 48 included in analysis. Tailored group: 47 exposed, 46 included in analysis.
16-ii) Primary analysis should be intent-to-treat Primary analysis should be intent-to-treat, secondary analyses could include comparing only "users", with the appropriate caveats that this is no longer a randomized sample (see 18-i).
subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 16-ii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 17a) For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval) Does your paper address CONSORT subitem 17a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 17a-i) Presentation of process outcomes such as metrics of use and intensity of use In addition to primary/secondary (clinical) outcomes, the presentation of process outcomes such as metrics of use and intensity of use (dose, exposure) and their operational definitions is critical. This does not only refer to metrics of attrition (13-b) (often a binary variable), but also to more continuous exposure metrics such as "average session length". These must be accompanied by a technical description how a metric like a "session" is defined (e.g., timeout after idle time) [1] (report under item 6a).
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 17a-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
Our primary outcome was care planning time. 8 of 211 subjects did not have a valid time due to technical difficulty and real life interruption. All other subjects were included in our analysis. We judge these subjects to be missing at random based on the circumstances. Given the small percentage (<5%), they should have minimal impact of effect estimation even if there was deviation from missing-at-randomness.
See Tables 3 and 4 in manuscript.
Not applicable given that the intervention was tailoring, done before subjects started the care planning.
17b) For binary outcomes, presentation of both absolute and relative effect sizes is recommended Does your paper address CONSORT subitem 17b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
18) Results of any other analyses performed, including subgroup analyses and adjusted analyses, distinguishing pre-specified from exploratory Does your paper address CONSORT subitem 18? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer

18-i) Subgroup analysis of comparing only users
A subgroup analysis of comparing only users is not uncommon in ehealth trials, but if done, it must be stressed that this is a self-selected sample and no longer an unbiased sample from a randomized trial (see 16-iii).
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem 18-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
19) All important harms or unintended effects in each group (for specific guidance see CONSORT for harms) We did not have any binary outcomes.
See Tables 4-6 for additional analysis.
Does your paper address CONSORT subitem 19? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
subitem not at all important essential 1 2 3 4 5 19-i) Include privacy breaches, technical problems Include privacy breaches, technical problems. This does not only include physical "harm" to participants, but also incidents such as perceived or real privacy breaches [1], technical problems, and other unexpected/unintended incidents. "Unintended effects" also includes unintended positive effects [2].
Does your paper address subitem 19-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
subitem not at all important essential 1 2 3 4 5 19-ii) Include qualitative feedback from participants or observations from staff/researchers Include qualitative feedback from participants or observations from staff/researchers, if available, on strengths and shortcomings of the application, especially if they point to unintended/ unexpected effects or uses. This includes (if available) reasons for why people did or did not use the application as intended by the developers.
Does your paper address subitem 19-ii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer DISCUSSION 22) Interpretation consistent with results, balancing benefits and harms, and considering other relevant evidence NPT: In addition, take into account the choice of the comparator, lack of or partial blinding, and unequal expertise of care providers or centers in each group No harm or unintended effects to report. None to report.
None to report.
subitem not at all important essential 1 2 3 4 5 22-i) Restate study questions and summarize the answers suggested by the data, starting with primary outcomes and process outcomes (use) Restate study questions and summarize the answers suggested by the data, starting with primary outcomes and process outcomes (use).
Does your paper address subitem 22-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

20-i) Typical limitations in ehealth trials
Typical limitations in ehealth trials: Participants in ehealth trials are rarely blinded. Ehealth trials often look at a multiplicity of outcomes, increasing risk for a Type I error. Discuss biases due to non-use of the intervention/usability issues, biases through informed consent procedures, unexpected events.
"Currently over 96% of general acute care hospitals use electronic health records [42] with nurses spending an increasing proportion of their time documenting their care in EHRs. Hospitals with EHRs with poorer usability have a significantly higher odds of nurse burnout, intention to leave, higher odds of inpatient mortality and 30-day readmission compared to those with better usability.
[43] It is therefore critically important to identify novel technical solutions for making EHRs and its various components more usable for nurses and other clinicians. In addition, these technical solutions should be tested first using simulation methods to reduce potential for patient harm and with an adequately powered diverse sample of nurses from across the country to assess effectiveness. We addressed these needs through the design and testing of a tailored EHR CDS in which RNs were assigned different CDS formats based on their graph literacy. This innovative simulation study was conducted virtually with a diverse national sample of 203 nurses. Conducting the study virtually was feasible and greatly lowered the cost and time commitment needed for travel, multiple IRB approval etc. associated with a typical multi-site study. We compared four CDS format groups for effects of the time spent on care planning and adoption of the recommendations. Participants adopted nearly 9 out of 10 offered CDS recommendations across the four CDS format groups. We expected the tailored CDS group to outperform the non-tailored CDS groups on adoption and time spent on care planning, but the study findings did "Given that adoption of recommendations was near 90% across all groups, there may be a ceiling effect on adoption rates, limiting variation across groups. If adoption rates are similar when studied in clinical practice when the CDS is implemented in clinical practice, patient care outcomes would be expected to be optimized because the CDS statements were derived from nursing care associated with better patient outcomes (cite aim studies from pilot R01). Future research in clinical setting measuring real patient outcomes is needed to validate prior findings with study designs supporting causal inferences. Research on prevalence of potentially flawed CDS recommendations, or higher risk recommendations (e.g., early mobility for patients on ECMO) and associated unintended consequences is also needed to understand whether RNs would have higher adoption rates with tailored formats." "In recent years, increasing attention has been brought to documentation burden and its negative impact on health care. While our study did not demonstrate significant differences in mental load across different CDS formats, study with a larger sample size and conducted in a real-world clinical settings (as opposed to experimental settings), where baseline stress is already high, is needed to provide more insight on the effect of CDS formats on nurse workload." Does your paper address subitem 20-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 21) Generalisability (external validity, applicability) of the trial findings NPT: External validity of the trial findings according to the intervention, comparators, patients, and care providers or centers involved in the trial subitem not at all important essential 1 2 3 4 5 21-i) Generalizability to other populations Generalizability to other populations: In particular, discuss generalizability to a general Internet population, outside of a RCT setting, and general patient population, including applicability of the study results for other organizations Does your paper address subitem 21-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
subitem not at all important essential 1 2 3 4 5 21-ii) Discuss if there were elements in the RCT that would be different in a routine application setting Discuss if there were elements in the RCT that would be different in a routine application setting (e.g., prompts/reminders, more human involvement, training sessions or other cointerventions) and what impact the omission of these elements could have on use, adoption, or outcomes if the intervention is applied outside of a RCT setting.
These concerns were not a major issue for us.
"Our study is not without limitations that should be considered as study findings are interpreted. The overall response rate was low and therefore there might have been a selective bias. The participating RNs may be atypical in their interest in research in general and health information technology in particular. The study was conducted during the COIVD-19 pandemic. The very high work load and stress placed on RNs might have an impact on the response rate and on the representativeness of the participating RNs. Testing novel EHR features in a simulation environment provides important insight, but is only one step towards improving EHR in clinical setting." Does your paper address subitem 21-ii? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer

23) Registration number and name of trial registry
Does your paper address CONSORT subitem 23? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer 24) Where the full trial protocol can be accessed, if available Does your paper address CONSORT subitem 24? * Cite a Multimedia Appendix, other reference, or copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Not applicable given the negative outcome.
This study did not involve patients and was NOT a clinical trial thus it was not registered under clinicaltrials.gov.
The trial protocol was described in the following article, "Dunn Lopez, K., Yao, Y., Cho, H., Dos Santos, F.C., Madanola, O., Bjarnadottir, R. Maciera, T. G. R., Garcia, A.L., Priola, K.J.B., Wolf, J., Bian, J., Wilkie, D.J. & Keenan, G.M. (In press, 2022) Conducting a representative national randomized control trial of tailored clinical decision support for nurses remotely: Methods and implications. Contemporary Clinical Trials." 25) Sources of funding and other support (such as supply of drugs), role of funders Does your paper address CONSORT subitem 25? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer X27) Conflicts of Interest (not a CONSORT item) X27-i) State the relation of the study team towards the system being evaluated In addition to the usual declaration of interests (financial or otherwise), also state the relation of the study team towards the system being evaluated, i.e., state if the authors/evaluators are distinct from or identical with the developers/sponsors of the intervention.
subitem not at all important essential 1 2 3 4 5 Does your paper address subitem X27-i? Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study

Your answer
About the CONSORT EHEALTH checklist As a result of using this checklist, did you make changes in your manuscript? * yes, major changes yes, minor changes no What were the most important changes you made as a result of using this checklist?
This study was funded by NIH.
There was no conflict of interest.

Abstract.
How much time did you spend on going through the checklist INCLUDING making changes in your manuscript *

Your answer
As a result of using this checklist, do you think your manuscript has improved? * yes no other: Would you like to become involved in the CONSORT EHEALTH group? This would involve for example becoming involved in participating in a workshop and writing an "Explanation and Elaboration" document yes no other: Any other comments or questions on CONSORT EHEALTH STOP -Save this form as PDF before you click submit To generate a record that you filled in this form, we recommend to generate a PDF of this page (on a Mac, simply select "print" and then select "print as PDF") before you submit it.
When you submit your (revised) paper to JMIR, please upload the PDF as supplementary file.
Don't worry if some text in the textboxes is cut off, as we still have the complete information in our database. Thank you! Final step: Click submit ! Click submit so we have your answers in our database! 10 hours.
Happy to participate in activities that involve my expertise and research methods.