Improving Skin Cancer Diagnostics Through a Mobile App With a Large Interactive Image Repository: Randomized Controlled Trial

Background Skin cancer diagnostics is challenging, and mastery requires extended periods of dedicated practice. Objective The aim of the study was to determine if self-paced pattern recognition training in skin cancer diagnostics with clinical and dermoscopic images of skin lesions using a large-scale interactive image repository (LIIR) with patient cases improves primary care physicians’ (PCPs’) diagnostic skills and confidence. Methods A total of 115 PCPs were randomized (allocation ratio 3:1) to receive or not receive self-paced pattern recognition training in skin cancer diagnostics using an LIIR with patient cases through a quiz-based smartphone app during an 8-day period. The participants’ ability to diagnose skin cancer was evaluated using a 12-item multiple-choice questionnaire prior to and 8 days after the educational intervention period. Their thoughts on the use of dermoscopy were assessed using a study-specific questionnaire. A learning curve was calculated through the analysis of data from the mobile app. Results On average, participants in the intervention group spent 2 hours 26 minutes quizzing digital patient cases and 41 minutes reading the educational material. They had an average preintervention multiple choice questionnaire score of 52.0% of correct answers, which increased to 66.4% on the postintervention test; a statistically significant improvement of 14.3 percentage points (P<.001; 95% CI 9.8-18.9) with intention-to-treat analysis. Analysis of participants who received the intervention as per protocol (500 patient cases in 8 days) showed an average increase of 16.7 percentage points (P<.001; 95% CI 11.3-22.0) from 53.9% to 70.5%. Their overall ability to correctly recognize malignant lesions in the LIIR patient cases improved over the intervention period by 6.6 percentage points from 67.1% (95% CI 65.2-69.3) to 73.7% (95% CI 72.5-75.0) and their ability to set the correct diagnosis improved by 10.5 percentage points from 42.5% (95% CI 40.2%-44.8%) to 53.0% (95% CI 51.3-54.9). The diagnostic confidence of participants in the intervention group increased on a scale from 1 to 4 by 32.9% from 1.6 to 2.1 (P<.001). Participants in the control group did not increase their postintervention score or their diagnostic confidence during the same period. Conclusions Self-paced pattern recognition training in skin cancer diagnostics through the use of a digital LIIR with patient cases delivered by a quiz-based mobile app improves the diagnostic accuracy of PCPs. Trial Registration ClinicalTrials.gov NCT05661370; https://classic.clinicaltrials.gov/ct2/show/NCT05661370


1a-i) Identify the mode of delivery in the title
Identify the mode of delivery. Preferably use "web-based" and/or "mobile" and/or "electronic game" in the title. Avoid ambiguous terms like "online", "virtual", "interactive". Use "Internet-based" only if Intervention includes non-web-based Internet components (e.g. email), use "computer-based" or "electronic" only if o ine products are used. Use "virtual" only in the context of "virtual reality" (3-D worlds). Use "online" only in the context of "online support groups". Complement or substitute product names with broader terms for the class of products (such as "mobile" or "smart phone" instead of "iphone"), especially if the application runs on different platforms. Does your paper address subitem 1a-i? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Subitem 1A-i is adressed in the title and several places in the manuscript.

1a-ii) Non-web-based components or important co-interventions in title
Mention non-web-based components or important co-interventions in title, if any (e.g., "with telephone support").
C l e a r s e l e c t i o n Does your paper address subitem 1a-ii?
Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Co-intervention does not play a big role in the study. The co-intervention of reminder-emails is mentioned in the manuscript: "They were sent email reminders on days four, seven, and eight. After the eight days, they were told to abstain from using the application for eight days (washout period)" subitem not at all important 1 2 3 4 5 essential 1b) ABSTRACT: Structured summary of trial design, methods, results, and conclusions NPT extension: Description of experimental treatment, comparator, care providers, centers, and blinding status.
1a-iii) Primary condition or target group in the title Mention primary condition or target group in the title, if any (e.g., "for children with Type I Diabetes") Example: A Web-based and Mobile Intervention with Telephone Support for Children with Type I Diabetes: Randomized Controlled Trial C l e a r s e l e c t i o n Does your paper address subitem 1a-iii? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study It could be argued that the title should include the study population (general practictioners or primary care physicians). We've decided not to include this item in the title, as there are so many variations to the interpretation and definition of general practitioner across the world. It is mentioned in the abstract several times.
subitem not at all important Does your paper address subitem 1b-i? * Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "115 PCPs were randomized (allocation ratio 3:1) to receive or not receive self-paced pattern recognition training in skin cancer diagnostics using a LIIR with patient cases through a quiz-based smartphone application during an eight-day period. " subitem not at all important 1 2 3 4 5 essential 1b-ii) Level of human involvement in the METHODS section of the ABSTRACT Clarify the level of human involvement in the abstract, e.g., use phrases like "fully automated" vs. "therapist/nurse/care provider/physician-assisted" (mention number and expertise of providers involved, if any). (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) C l e a r s e l e c t i o n Does your paper address subitem 1b-ii?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not directly. It explains how the mobile application functions, and the explanation includes no mentions of interaction with other humans. The term "fully automated" could be applied, but would distract from the general thought and understanding of a quiz-based mobile application.
subitem not at all important 1 2 3 4 5 essential 1b-iii) Open vs. closed, web-based (self-assessment) vs. face-to-face assessments in the METHODS section of the ABSTRACT Mention how participants were recruited (online vs. o ine), e.g., from an open access website or from a clinic or a closed online user group (closed usergroup trial), and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment). Clearly say if outcomes were self-assessed through questionnaires (as common in web-based trials). Note: In traditional o ine trials, an open trial (open-label trial) is a type of clinical trial in which both the researchers and participants know which treatment is being administered. To avoid confusion, use "blinded" or "unblinded" to indicated the level of blinding instead of "open", as "open" in web-based trials usually refers to "open access" (i.e. participants can self-enrol). (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) C l e a r s e l e c t i o n Does your paper address subitem 1b-iii?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study No. Recruitment is not part of the abstract, focusing on the intervention, outcomes and results. The recruitment method and population is elaborated in the main manuscript.
subitem not at all important 1 2 3 4 5 essential 1b-iv) RESULTS section in abstract must contain use data Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study The conclusion part of the abstract in general terms address how the intervention affects the population on the primary and secondary outcome peramiters: "Self-paced pattern recognition training in skin cancer diagnostics through the use of a digital large-scale interactive image repository with patient cases delivered by a quiz-based mobile application improves the diagnostic accuracy of primary care physicians." subitem not at all important 1 2 3 4 5 essential 2a-i) Problem and the type of system/solution . This has never been tested among PCPs." The study uses a non-intervention control as comparator.

METHODS
3a) Description of trial design (such as parallel, factorial) including allocation ratio 3b) Important changes to methods after trial commencement (such as eligibility criteria), with reasons Does your paper address CONSORT subitem 2b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study The description of subitem 2b is mission from the form.
But yes, it is part of the introduction: "With this study, we aimed to examine if self-paced training in skin cancer diagnostics using a LIIR with patient cases improves PCPs' diagnostic skills. In addition, we will investigate their ability to set the correct diagnosis (diagnostic accuracy) and to correctly classify digital patient cases as benign or malignant, their change in diagnostic confidence, and measure their time spent on the intervention." Does your paper address CONSORT subitem 3a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "This randomized controlled trial used block randomization in blocks of 4, using an online randomizer [32] to generate a random number sequence of numbers 1 through 4. Participants were allocated continuously and sequentially as they signed up to either Intervention or Control with a 3:1 ratio" subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 3b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study There were no changes to methods after trial commencement in this study. But the question is highly relevant.

3b-i) Bug fixes, Downtimes, Content Changes
Bug xes, Downtimes, Content Changes: ehealth systems are often dynamic systems. A description of changes to methods therefore also includes important changes made on the intervention or comparator during the trial (e.g., major bug xes or changes in the functionality or content) (5-iii) and other "unexpected events" that may have in uenced study design such as staff changes, system failures/downtimes, etc. [2].
C l e a r s e l e c t i o n Does your paper address subitem 3b-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study There were no downtimes, content changes or similar during the trial. "The Dermloop Lean application underwent no changes, updates or bug fixes and there was no downtime during the trial period." 4a) Eligibility criteria for participants subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 4a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "Eligible PCPs (doctors currently working in the primary care sector) were recruited"...

4a-i) Computer / Internet literacy
Computer / Internet literacy is often an implicit "de facto" eligibility criterion -this should be explicitly clari ed. Does your paper address subitem 4a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study The recruitment process is explained in detail. Several steps involve use of internet, computers or smartphones, filling out an online survey etc. The participants weren't particularly tech savvy, but the recruitment method will have preselected those participants that were able to fulfill all steps in the process. This is not explained in detail in the manuscript, but can be read from the recruitment process.
4a-ii) Open vs. closed, web-based vs. face-to-face assessments: Open vs. closed, web-based vs. face-to-face assessments: Mention how participants were recruited (online vs. o ine), e.g., from an open access website or from a clinic, and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment), i.e., to what degree got the study team to know the participant. In online-only trials, clarify if participants were quasi-anonymous and whether having multiple identities was possible or whether technical or logistical measures (e.g., cookies, email con rmation, phone calls) were used to detect/prevent these. Does your paper address subitem 4a-ii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "Eligible PCPs (doctors currently working in the primary care sector) were recruited at a conference in November 2021 (Laegedage) in Denmark by the speakers at three skin cancer and melanoma sessions. Interested physicians scanned a QR code and signed up to receive information material and an invitation to participate in the study by email, including a link to an online survey [Google Forms, Google Ireland Limited, 2022] which contained a consent form. The survey also included questions about their experience with and use of dermoscopy, including their confidence in their diagnostic abilities on a scale from 1 (low) to 4 (high), and ended with a skin cancer multiple choice questionnaire (MCQ) including 12 patient cases from a list of 25 skin lesion cases with previously established validity evidence [1]. "

4a-iii) Information giving during recruitment
Information given during recruitment. Specify how participants were briefed for recruitment and in the informed consent procedures (e.g., publish the informed consent documentation as appendix, see also item X26), as this information may have an effect on user self-selection, user expectation and may also bias results. Does your paper address subitem 4a-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Participants in the intervention group were invited to one of several online initiation meetings where they were instructed on how to download, install, sign up, and access the LIIR through a quiz-based smartphone application for practicing skin cancer diagnostics. Participants could also download and install the application independently using a pdfguide. After installation, the participants in the intervention group were given eight days of access from the day of their sign-up in the application, in which they were asked to diagnose 500 digital patient cases. They were sent email reminders on days four, seven, and eight. After the eight days, they were told to abstain from using the application for eight days " Does your paper address CONSORT subitem 4b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "No measurements were taken to blind the Principal Investigator when receiving questionnaire responses and MCQ test results or when performing the statistical analysis comparing the two groups." subitem not at all important 1 2 3 4 5 essential 4b-i) Report if outcomes were (self-)assessed through online questionnaires Clearly report if outcomes were (self-)assessed through online questionnaires (as common in web-based trials) or otherwise.
C l e a r s e l e c t i o n Does your paper address subitem 4b-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Primary outcomes were not self assessed. Some secondary outcomes reported in the article were self-assessed, which is described: "Diagnostic confidence The participants in both groups initially had identical relatively low confidence in their diagnostic ability to diagnose skin lesions using a dermoscope of 1.6 on a scale from 1 (low) to 4 (high), which increased significantly for participants in the intervention group by 32.9% to 2.1 (p<.001) after the intervention, but not for the control group (p=0.23)." subitem not at all important 1 2 3 4 5 essential 5) The interventions for each group with sufficient details to allow replication, including how and when they were actually administered 4b-ii) Report how institutional affiliations are displayed Report how institutional a liations are displayed to potential participants [on ehealth media], as a liations with prestigious hospitals or universities may affect volunteer rates, use, and reactions with regards to an intervention.(Not a required item -describe only if this may bias results)

Does your paper address subitem 4b-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Your answer subitem not at all important 1 2 3 4 5 essential 5-i) Mention names, credential, affiliations of the developers, sponsors, and owners Mention names, credential, a liations of the developers, sponsors, and owners [6] (if authors/evaluators are owners or developer of the software, this needs to be declared in a "Con ict of interest" section or mentioned elsewhere in the manuscript).
C l e a r s e l e c t i o n

Does your paper address subitem 5-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. All author affiliations are reported as well as conflicts of interest: "Conflicts of Interest NKT is the Chief Executive Officer of the software company Melatech ApS, which has developed the mobile application Dermloop Learn. The remainder of the authors have no conflicts of interest." subitem not at all important 1 2 3 4 5 essential 5-ii) Describe the history/development process Describe the history/development process of the application and previous formative evaluations (e.g., focus groups, usability testing), as these will have an impact on adoption/use rates and help with interpreting results.
C l e a r s e l e c t i o n

Does your paper address subitem 5-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study In part. The article references another article using the intervention, in which the development is described in more detail.

5-iii) Revisions and updating
Revisions and updating. Clearly mention the date and/or version number of the application/intervention (and comparator, if applicable) evaluated, or describe whether the intervention underwent major changes during the evaluation process, or whether the development and/or content was "frozen" during the trial. Describe dynamic components such as news feeds or changing content which may have an impact on the replicability of the intervention (for unexpected events see item 3b).
C l e a r s e l e c t i o n Does your paper address subitem 5-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study

5-iv) Quality assurance methods
Provide information on quality assurance methods to ensure accuracy and quality of information provided [1], if applicable.

Does your paper address subitem 5-iv?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study This item is not applicable to this study.
subitem not at all important 1 2 3 4 5 essential 5-v) Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing flowcharts of the algorithms used Ensure replicability by publishing the source code, and/or providing screenshots/screencapture video, and/or providing owcharts of the algorithms used. Replicability (i.e., other researchers should in principle be able to replicate the study) is a hallmark of scienti c reporting.
C l e a r s e l e c t i o n

Does your paper address subitem 5-v?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes, the figure in the manuscript shows the use-flow and interface, and the text and tables describe what diagnoses were available at the time of testing.
"The participants trained pattern recognition (quizzing) with sessions containing 10 cases selected from the library. One case was presented at a time with a clinical image, dermoscopic image, and the lesion's location. The participant selected benign (and picked the suspected diagnosis from among nevus, seborrheic keratosis/solar lentigo, dermatofibroma, or hemangioma) or malignant (and picked the suspected diagnosis from among melanoma, basal cell carcinoma, or squamous cell carcinoma). Immediately after answering a case, the user got feedback on whether the answer was correct or not, what the correct answer was, and an option of being taken to a written learning module on the correct and incorrect answer, respectively. " subitem not at all important 1 2 3 4 5 essential

5-vi) Digital preservation
Digital preservation: Provide the URL of the application, but as the intervention is likely to change or disappear over the course of the years; also make sure the intervention is archived (Internet Archive, webcitation.org, and/or publishing the source code or screenshots/videos alongside the article). As pages behind login screens cannot be archived, consider creating demo pages which are accessible without login.
C l e a r s e l e c t i o n Does your paper address subitem 5-vi?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study The beta version is no longer accessible, but the newest version is freely available. URL included in the references.

5-vii) Access
Access: Describe how participants accessed the application, in what setting/context, if they had to pay (or were paid) or not, whether they had to be a member of speci c group. If known, describe how participants obtained "access to the platform and Internet" [1]. To ensure access for editors/reviewers/readers, consider to provide a "backdoor" login account or demo mode for reviewers/readers to explore the application (also important for archiving purposes, see vi).
C l e a r s e l e c t i o n Does your paper address subitem 5-vii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. Yet, a detailed description of how the participants accessed the beta version of the app by installing a web-page (progressive web app) as an app on their phone is not included, as the focus of the article is not the digital development or access, but the use of and results of use of the application.
"A total of 12 participants did not manage to download and install the mobile application, which was yet to be released to the general public at the time. " "The participants were given no compensation for their time spent, which was done primarily in their spare time. " subitem not at all important 1 2 3 4 5 essential 5-viii) Mode of delivery, features/functionalities/components of the intervention and comparator, and the theoretical framework Does your paper address subitem 5-viii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes.
The text and figures describe many of these elements of use of the application " Each digital patient case included the age and gender of the patient, a clinical and a dermoscopic image of the lesion, and its location on a 3D avatar; all of which are referred to as a "case". See Figure 2 for an overview and example. The participants trained pattern recognition (quizzing) with sessions containing 10 cases selected from the library. One case was presented at a time with a clinical image, dermoscopic image, and the lesion's location. The participant selected benign (and picked the suspected diagnosis from among nevus, seborrheic keratosis/solar lentigo, dermatofibroma, or hemangioma) or malignant (and picked the suspected diagnosis from among melanoma, basal cell carcinoma, or squamous cell carcinoma). Immediately after answering a case, the user got feedback on whether the answer was correct or not, what the correct answer was, and an option of being taken to a written learning module on the correct and incorrect answer, respectively. The smartphone application has written learning modules on 36 diagnoses and sub-diagnoses corresponding to the diagnoses of the cases contained in the LIIR. Each written learning module has an introductory section followed by sections on histopathology, clinical presentation, dermoscopic features, and differential diagnoses to the diagnosis, all including illustrations and/or examples from the LIIR. As the user answered cases, the application compiled a list of the diagnoses for which the individual user had the most difficulty giving correct answers. The list was shown on the front page of the application (Overview page in Figure 2), nudging the user towards reading the corresponding written learning modules. The application also tracked how many cases each user had diagnosed, their answer to each case, how long they spent with each case, what written modules they opened, and how long they spent reading each time they opened a written module. " subitem not at all important 1 2 3 4 5 essential

5-ix) Describe use parameters
Describe use parameters (e.g., intended "doses" and optimal timing for use). Clarify what instructions or recommendations were given to the user, e.g., regarding timing, frequency, heaviness of use, if any, or was the intervention used ad libitum.
C l e a r s e l e c t i o n Does your paper address subitem 5-ix?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "After installation, the participants in the intervention group were given eight days of access from the day of their sign-up in the application, in which they were asked to diagnose 500 digital patient cases. They were sent email reminders on days four, seven, and eight. After the eight days, they were told to abstain from using the application for eight days (washout period) before answering a final skin cancer MCQ with 12 new cases." subitem not at all important 1 2 3 4 5 essential

5-x) Clarify the level of human involvement
Clarify the level of human involvement (care providers or health professionals, also technical assistance) in the e-intervention or as co-intervention (detail number and expertise of professionals involved, if any, as well as "type of assistance offered, the timing and frequency of the support, how it is initiated, and the medium by which the assistance is delivered". It may be necessary to distinguish between the level of human involvement required for the trial, and the level of human involvement required for a routine application outside of a RCT setting (discuss under item 21 -generalizability).
C l e a r s e l e c t i o n Does your paper address subitem 5-x?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "Participants in the intervention group were invited to one of several online initiation meetings where they were instructed on how to download, install, sign up, and access the LIIR through a quiz-based smartphone application for practicing skin cancer diagnostics. " ... "They were sent email reminders on days four, seven, and eight. After the eight days, they were told to abstain from using the application for eight days (washout period) before answering a final skin cancer MCQ with 12 new cases." subitem not at all important 1 2 3 4 5 essential 5-xi) Report any prompts/reminders used Report any prompts/reminders used: Clarify if there were prompts (letters, emails, phone calls, SMS) to use the application, what triggered them, frequency etc. It may be necessary to distinguish between the level of prompts/reminders required for the trial, and the level of prompts/reminders for a routine application outside of a RCT setting (discuss under item 21 -generalizability).
C l e a r s e l e c t i o n Does your paper address subitem 5-xi? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "They were sent email reminders on days four, seven, and eight. After the eight days, they were told to abstain from using the application for eight days (washout period) before answering a final skin cancer MCQ with 12 new cases." subitem not at all important 1 2 3 4 5 essential 6a) Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed 5-xii) Describe any co-interventions (incl. training/support) Describe any co-interventions (incl. training/support): Clearly state any interventions that are provided in addition to the targeted eHealth intervention, as ehealth intervention may not be designed as stand-alone intervention. This includes training sessions and support [1]. It may be necessary to distinguish between the level of training required for the trial, and the level of training for a routine application outside of a RCT setting (discuss under item 21 -generalizability. C l e a r s e l e c t i o n Does your paper address subitem 5-xii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. This intervention is designed to be usable as a stand-alone intervention. This is not stated specifically, but can be read from the trial setup.
subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 6a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "The trial's primary outcome was the participants' score on a skin cancer MCQ before and after the intervention, with both per-protocol and intention-to-treat analysis. Secondary outcomes included the progression of the participants' ability to correctly diagnose and classify digital patient cases in the LIIR across the intervention period, descriptive analysis of which diagnoses were most commonly misclassified and misdiagnosed, chance in the participants' diagnostic confidence, and the average time spent training." 6a-i) Online questionnaires: describe if they were validated for online use and apply CHERRIES items to describe how the questionnaires were designed/deployed If outcomes were obtained through online questionnaires, describe if they were validated for online use and apply CHERRIES items to describe how the questionnaires were designed/deployed [9]. Does your paper address subitem 6a-i?

Copy and paste relevant sections from manuscript text
Yes and no. The use of online forms is described, but these were not validated. Nor has CHERRIES been applied systematically in the design.
6a-ii) Describe whether and how "use" (including intensity of use/dosage) was defined/measured/monitored Describe whether and how "use" (including intensity of use/dosage) was de ned/measured/monitored (logins, log le analysis, etc.). Use/adoption metrics are important process outcomes that should be reported in any ehealth trial. Does your paper address subitem 6a-ii?
Copy and paste relevant sections from manuscript text Yes. Use of the platform to diagnose cases: "The intervention group completed 467 cases on average (median of 505, range 0-1679), and 54 participants completed 500 cases (as per protocol) or more and were included in the Per-Protocol analysis. All 33 participants from the control group finished the final MCQ. The time from the initial to the final MCQ was 21 and 19 days for the intervention and control groups, respectively." Use of the platform in terms of time: "Time spent quizzing and reading During the eight-day intervention period, participants from the intervention group, on average, spent 2 hours and 26 minutes (ranging from 0 minutes to 8 hours and 35 minutes) practicing pattern recognition (quizzing) and 41 minutes (ranging from 0 minutes to 3 hours 23 minutes) reading the written educational modules included in the app. The majority of the participants' reading activity was done in the beginning of the period and fell drastically after the first 100 cases (solid blue line in Figure 4). The average time spent diagnosing each case decreased from 36 to 28 seconds (solid red line in Figure 4)." 6a-iii) Describe whether, how, and when qualitative feedback from participants was obtained Describe whether, how, and when qualitative feedback from participants was obtained (e.g., through emails, feedback forms, interviews, focus groups).
C l e a r s e l e c t i o n 6b) Any changes to trial outcomes after the trial commenced, with reasons 7a) How sample size was determined NPT: When applicable, details of whether and how the clustering by care provides or centers was addressed Does your paper address subitem 6a-iii?
Copy and paste relevant sections from manuscript text Qualitative feedback was obtained as part of the final quiz, delivered through a google forms questionnaire. This is not included in the manuscript (despite being very positive), as we decided that it was not relevant to the main story of the article and its findings.
Does your paper address CONSORT subitem 6b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study No changes to the trial outcomes after trial commenced.
subitem not at all important 1 2 3 4 5 essential 7b) When applicable, explanation of any interim analyses and stopping guidelines 7a-i) Describe whether and how expected attrition was taken into account when calculating the sample size Describe whether and how expected attrition was taken into account when calculating the sample size.
C l e a r s e l e c t i o n

Does your paper address subitem 7a-i?
Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Sample-size calculations were done, and then enrollment continued past that to ensure that a even a 20% dropout could be tolerated. The sample-size calculations are described under Methods -> Statistical Analysis.
The enrollment period continued past the required number of participants as per popular demand and was a fixed period set by the availability of the Principal Investigator to do onboarding/recruitment.

8a) Method used to generate the random allocation sequence
NPT: When applicable, how care providers were allocated to each trial group 8b) Type of randomisation; details of any restriction (such as blocking and block size) Does your paper address CONSORT subitem 7b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not Applicable for this trial.
Does your paper address CONSORT subitem 8a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "This randomized controlled trial used block randomization in blocks of 4, using an online randomizer [32] to generate a random number sequence of numbers 1 through 4. Participants were allocated continuously and sequentially as they signed up to either Intervention or Control with a 3:1 ratio..." 9) Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned 10) Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions Does your paper address CONSORT subitem 8b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "This randomized controlled trial used block randomization in blocks of 4, using an online randomizer [32] to generate a random number sequence of numbers 1 through 4. Participants were allocated continuously and sequentially as they signed up to either Intervention or Control with a 3:1 ratio..." Does your paper address CONSORT subitem 9? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "This randomized controlled trial used block randomization in blocks of 4, using an online randomizer [32] to generate a random number sequence of numbers 1 through 4. Participants were allocated continuously and sequentially as they signed up to either Intervention or Control with a 3:1 ratio..." 11a) If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how NPT: Whether or not administering co-interventions were blinded to group assignment subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 10? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study The principal investigator. Since participants were enrolled continuously and sequentially there was no risk of bias by letting the PI make the allocation sequence.
"Participants were allocated continuously and sequentially as they signed up to either Intervention or Control with a 3:1 ratio..."

11a-i) Specify who was blinded, and who wasn't
Specify who was blinded, and who wasn't. Usually, in web-based trials it is not possible to blind the participants [1, 3] (this should be clearly acknowledged), but it may be possible to blind outcome assessors, those doing data analysis or those administering cointerventions (if any). Does your paper address subitem 11a-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "Participants were unaware of their allocation until they had answered the initial questionnaire and MCQ. No measurements were taken to blind the Principal Investigator when receiving questionnaire responses and MCQ test results or when performing the statistical analysis comparing the two groups." 11a-ii) Discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator" Informed consent procedures (4a-ii) can create biases and certain expectations -discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator". 12a) Statistical methods used to compare groups for primary and secondary outcomes NPT: When applicable, details of whether and how the clustering by care providers or centers was addressed Does your paper address subitem 11a-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes.
There was no blinding as to who were in the intervention group and who were control. They were specifically told this upon randomization.
Does your paper address CONSORT subitem 11b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study No. It is not applicable as the trial is "Something vs. nothing".
subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 12a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes.
"Based on the methods of the similar and recent study by our research group [31] using Generalized Estimating Equations, learning curves of the participants' probability to diagnose cases correctly were expressed as linear splines with a single knot at 100 cases using a logistic regression model with random intercept with correct diagnosis or not as outcome. A similar model was made with correct classification of each case as malignant or benign as the outcome. The participants' time spent reading and quizzing was summed separately for each 100-case period. MCQ scores for both groups before and after the study period were compared using Welch's Two Sample t-test."

12a-i) Imputation techniques to deal with attrition / missing values
Imputation techniques to deal with attrition / missing values: Not all participants will use the intervention/comparator as intended and attrition is typically high in ehealth trials. Specify how participants who did not use the application or dropped out from the trial were treated in the statistical analysis (a complete case analysis is strongly discouraged, and simple imputation techniques such as LOCF may also be problematic [4]). Does your paper address subitem 12a-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes.
As the primary outcome was obtained at the end of the trial, dropouts that did not respond to the last quiz had no data to analyse.
In-system data from participants that did not respond to the last quiz were excluded.
"In this study, the intention-to-treat analysis included those individuals who participated and also answered the final questionnaire. The per-protocol analysis included those individuals who completed the stipulated 500 cases." Does your paper address CONSORT subitem 12b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes, the methods are described in the results section.
subitem not at all important

RESULTS
13a) For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome NPT: The number of care providers or centers performing the intervention in each group and the number of patients treated by each care provider in each center

X26-iii) Safety and security procedures
Safety and security procedures, incl. privacy considerations, and any steps taken to reduce the likelihood or detection of harm (e.g., education and training, availability of a hotline) C l e a r s e l e c t i o n

Does your paper address subitem X26-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not included in the paper. The participants signed "Terms of Use" in the application where their rights and the safety and security procedures were described.
13b) For each group, losses and exclusions after randomisation, together with reasons Does your paper address CONSORT subitem 13a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "A total of 279 people applied for more information about the study, 135 accepted the invitation and finished the initial questionnaire, of which seven were excluded as they were not doctors from the primary care sector. Initially, 96 were randomized to the intervention and 33 to the control group. From the intervention group, 84 subsequently set up a user profile granting them access to the smartphone app and its LIIR, of which 82 answered the final MCQ and were included in the Intention-To-Treat analysis (see CONSORT diagram in Figure 1). " Does your paper address CONSORT subitem 13b? (NOTE: Preferably, this is shown in a CONSORT flow diagram) * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "A total of 279 people applied for more information about the study, 135 accepted the invitation and finished the initial questionnaire, of which seven were excluded as they were not doctors from the primary care sector. Initially, 96 were randomized to the intervention and 33 to the control group. From the intervention group, 84 subsequently set up a user profile granting them access to the smartphone app and its LIIR, of which 82 answered the final MCQ and were included in the Intention-To-Treat analysis (see CONSORT diagram in Figure 1). " subitem not at all important Does your paper address subitem 13b-i?
Copy and paste relevant sections from the manuscript or cite the gure number if applicable (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes, it is shown in the CONSORT diagram.
subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 14a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "Eligible PCPs (doctors currently working in the primary care sector) were recruited at a conference in November 2021 (Laegedage) in Denmark by the speakers at three skin cancer and melanoma sessions. " 14a-i) Indicate if critical "secular events" fell into the study period Indicate if critical "secular events" fell into the study period, e.g., signi cant changes in Internet resources available or "changes in computer hardware or Internet delivery resources" C l e a r s e l e c t i o n Does your paper address subitem 14a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "Despite the trial period extending across the busy time before and during Christmas and New Year, the study had a high level of adherence." 14b) Why the trial ended or was stopped (early)

15) A table showing baseline demographic and clinical characteristics for each
group NPT: When applicable, a description of care providers (case volume, quali cation, expertise, etc.) and centers (volume) in each group Does your paper address CONSORT subitem 14b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable to this trial.
Does your paper address CONSORT subitem 15? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes, tables 1 and 2 shows participant demographics and their clinical characteristics.
subitem not at all important 1 2 3 4 5 essential 16) For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups 15-i) Report demographics associated with digital divide issues In ehealth trials it is particularly important to report demographics associated with digital divide issues, such as age, education, gender, social-economic status, computer/Internet/ehealth literacy of the participants, if known.
C l e a r s e l e c t i o n Does your paper address subitem 15-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. Table 1 and 2 shows some of these demographics, as well as their general use of dermoscopy -a pseudometric for tech "savvyness" in this population.
subitem not at all important 1 2 3 4 5 essential 16-i) Report multiple "denominators" and provide definitions Report multiple "denominators" and provide de nitions: Report N's (and effect sizes) "across a range of study participation [and use] thresholds" [1], e.g., N exposed, N consented, N used more than x times, N used more than y weeks, N participants "used" the intervention/comparator at speci c pre-de ned time points of interest (in absolute and relative numbers per group). Always clearly de ne "use" of the intervention.
C l e a r s e l e c t i o n Does your paper address subitem 16-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "Initially, 96 were randomized to the intervention and 33 to the control group. From the intervention group, 84 subsequently set up a user profile granting them access to the smartphone app and its LIIR, of which 82 answered the final MCQ and were included in the Intention-To-Treat analysis (see CONSORT diagram in Figure 1). The intervention group completed 467 cases on average (median of 505, range 0-1679), and 54 participants completed 500 cases (as per protocol) or more and were included in the Per-Protocol analysis. All 33 participants from the control group finished the final MCQ. The time from the initial to the final MCQ was 21 and 19 days for the intervention and control groups, respectively." And then the section about time spend as well.
subitem not at all important 1 2 3 4 5 essential 17a) For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval)

16-ii) Primary analysis should be intent-to-treat
Primary analysis should be intent-to-treat, secondary analyses could include comparing only "users", with the appropriate caveats that this is no longer a randomized sample (see 18-i).
C l e a r s e l e c t i o n Does your paper address subitem 16-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Agreed. "Intervention effect The average MCQ score of the PCPs in the intervention group (intention-to-treat analysis) improved from 52.0% (6.2 correct answers out of 12) to 66.4% (8.0 correct answers out of 12); an improvement of 14.3 percentage points [95% CI 9.8-18.9, p<.001]. Those participants who diagnosed 500 patient cases or more (per-protocol analysis) on average improved from 53.9% (6.5 correct answers out of 12) to 70.5% ( Does your paper address CONSORT subitem 17a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "Intervention effect The average MCQ score of the PCPs in the intervention group (intention-to-treat analysis) improved from 52.0% (6.2 correct answers out of 12) to 66.4% (8.0 correct answers out of 12); an improvement of 14.3 percentage points [95% CI 9.8-18.9, p<.001]. Those participants who diagnosed 500 patient cases or more (per-protocol analysis) on average improved from 53.9% (6.5 correct answers out of 12) to 70.5% (8.5 correct answers); an improvement of 16.6 percentage points [95% CI 11.3-22.0, p<.001]. "

17a-i) Presentation of process outcomes such as metrics of use and intensity of use
In addition to primary/secondary (clinical) outcomes, the presentation of process outcomes such as metrics of use and intensity of use (dose, exposure) and their operational de nitions is critical. This does not only refer to metrics of attrition (13-b) (often a binary variable), but also to more continuous exposure metrics such as "average session length". These must be accompanied by a technical description how a metric like a "session" is de ned (e.g., timeout after idle time) [1] (report under item 6a). Does your paper address subitem 17a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. "The intervention group completed 467 cases on average (median of 505, range 0-1679), and 54 participants completed 500 cases (as per protocol) or more and were included in the Per-Protocol analysis. All 33 participants from the control group finished the final MCQ. " "During the eight-day intervention period, participants from the intervention group, on average, spent 2 hours and 26 minutes (ranging from 0 minutes to 8 hours and 35 minutes) practicing pattern recognition (quizzing) and 41 minutes (ranging from 0 minutes to 3 hours 23 minutes) reading the written educational modules included in the app. The majority of the participants' reading activity was done in the beginning of the period and fell drastically after the first 100 cases (solid blue line in Figure 4). The average time spent diagnosing each case decreased from 36 to 28 seconds (solid red line in Figure 4)." Does your paper address CONSORT subitem 17b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study The outcomes are not binary.
subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 18? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Post-hoc and sub-group analyses are clearly described as such. For example: "A post-hoc analysis of the participant's initial MCQ score and their years of experience diagnosing skin lesions found no correlation, as shown in Figure 4. "

18-i) Subgroup analysis of comparing only users
A subgroup analysis of comparing only users is not uncommon in ehealth trials, but if done, it must be stressed that this is a self-selected sample and no longer an unbiased sample from a randomized trial (see 16-iii).
C l e a r s e l e c t i o n Does your paper address subitem 18-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Post-hoc and sub-group analyses are clearly described as such. For example: "A post-hoc analysis of the participant's initial MCQ score and their years of experience diagnosing skin lesions found no correlation, as shown in Figure 4. "

19) All important harms or unintended effects in each group
(for speci c guidance see CONSORT for harms) subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 19? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study None found during the trial.

19-i) Include privacy breaches, technical problems
Include privacy breaches, technical problems. This does not only include physical "harm" to participants, but also incidents such as perceived or real privacy breaches Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study This part is not included in the manuscript, but some has been collected in the post-trial questionnaire, which has led to changes to the new version of the App. These findings can be reported in the next articles on the subject, where they are better suited. Restate study questions and summarize the answers suggested by the data, starting with primary outcomes and process outcomes (use

20-i) Typical limitations in ehealth trials
Typical limitations in ehealth trials: Participants in ehealth trials are rarely blinded. Ehealth trials often look at a multiplicity of outcomes, increasing risk for a Type I error. Discuss biases due to non-use of the intervention/usability issues, biases through informed consent procedures, unexpected events.
C l e a r s e l e c t i o n Does your paper address subitem 20-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Partly. The article describes how the blinding process was done. There is very little risk of multiplicity of outcomes in this trial. The other risks of bias are not addressed in the manuscript, as many of them are not directly applicable.

21-i) Generalizability to other populations
Generalizability to other populations: In particular, discuss generalizability to a general Internet population, outside of a RCT setting, and general patient population, including applicability of the study results for other organizations C l e a r s e l e c t i o n Does your paper address subitem 21-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. This is specifically reported in the manuscript: "Limitations of this study include the recruitment method of inviting doctors participating in continuing education sessions about skin cancer. These doctors were perhaps more interested in or concerned about skin cancer diagnostics than the general population of PCPs. Evidence of this was that our participants had a relatively high initial diagnostic accuracy, most (84%) reported having access to a dermoscope in their clinic, and the percentage of participants who had received training (not including peer-to-peer and selfinitiated training) in dermoscopy was also relatively high (46%). The effect of the educational intervention would possibly be more distinct in the general PCP population, where a lack of specific training in skin cancer diagnostics is more common Discuss if there were elements in the RCT that would be different in a routine application setting (e.g., prompts/reminders, more human involvement, training sessions or other cointerventions) and what impact the omission of these elements could have on use, adoption, or outcomes if the intervention is applied outside of a RCT setting.
C l e a r s e l e c t i o n Does your paper address subitem 21-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes. This is specifically reported in the manuscript: "Limitations of this study include the recruitment method of inviting doctors participating in continuing education sessions about skin cancer. These doctors were perhaps more interested in or concerned about skin cancer diagnostics than the general population of PCPs. Evidence of this was that our participants had a relatively high initial diagnostic accuracy, most (84%) reported having access to a dermoscope in their clinic, and the percentage of participants who had received training (not including peer-to-peer and selfinitiated training) in dermoscopy was also relatively high (46%). The effect of the educational intervention would possibly be more distinct in the general PCP population, where a lack of specific training in skin cancer diagnostics is more common [22]. " 23) Registration number and name of trial registry 24) Where the full trial protocol can be accessed, if available 25) Sources of funding and other support (such as supply of drugs), role of funders X27) Conflicts of Interest (not a CONSORT item) Does your paper address CONSORT subitem 23? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes, these are included in the manuscript Does your paper address CONSORT subitem 24? * Cite a Multimedia Appendix, other reference, or copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes, these are included in the manuscript Does your paper address CONSORT subitem 25? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes, these are included in the manuscript subitem not at all important 1 2 3 4 5 essential About the CONSORT EHEALTH checklist yes, major changes yes, minor changes no X27-i) State the relation of the study team towards the system being evaluated In addition to the usual declaration of interests ( nancial or otherwise), also state the relation of the study team towards the system being evaluated, i.e., state if the authors/evaluators are distinct from or identical with the developers/sponsors of the intervention.
C l e a r s e l e c t i o n Does your paper address subitem X27-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes, these are included in the manuscript As a result of using this checklist, did you make changes in your manuscript? * yes no Other: yes no Other: What were the most important changes you made as a result of using this checklist?
The sentence about downtime or changes to the intervention during the intervention period. Simply because this is very relevant to another study we will be conducting.
How much time did you spend on going through the checklist INCLUDING making changes in your manuscript STOP -Save this form as PDF before you click submit To generate a record that you lled in this form, we recommend to generate a PDF of this page (on a Mac, simply select "print" and then select "print as PDF") before you submit it.
When you submit your (revised) paper to JMIR, please upload the PDF as supplementary le.
Don't worry if some text in the textboxes is cut off, as we still have the complete information in our database. Thank you!