A guideline for reporting experimental protocols in life sciences

Materials

We have analyzed: (i) guidelines for authors from journals publishing protocols (Giraldo, Garcia & Corcho, 2018b), (ii) our corpus of protocols (Giraldo, Garcia & Corcho, 2018a), (iii) a set of reporting structures proposed by minimal information projects available in the FairSharing catalog (McQuilton et al., 2016), and (iv) relevant biomedical ontologies available in BioPortal (Whetzel et al., 2011) and Ontobee (Xiang et al., 2011). Our analysis was carried out by a domain expert, Olga Giraldo; she is an expert in text mining and biomedical ontologies with over ten years of experience in laboratory techniques. All the documents were read, and then data elements, subject areas, materials (e.g., sample, kits, solutions, reagents, etc.), and workflow information were identified. Resulting from this activity we established a baseline terminology, common and non common data elements, as well as patterns in the description of the workflows (e.g., information describing the steps and the order for the execution of the workflow).

Methods for developing this guideline

Developing the guideline entailed a series of activities; these were organized in the following stages: (i) analysis of guidelines for authors, (ii) analysis of protocols, (iii) analysis of Minimum Information (MI) standards and ontologies, and (iv) evaluation of the data elements from our guideline. For a detailed representation of our workflow, see Fig. 1

Bibliographic data elements

From the guidelines for authors, the datum “author identifier” was not considered, nor was this data element found in the analyzed protocols. The “provenance” is proposed as “desirable information” in only two of the guidelines (Nature Protocols and Bio-protocols), as well as “updates of the protocol” (Cold Spring Harbor Protocols and Bio-protocols). A total of 72.5% (29) of the protocols available in our Bio-protocols collection and 61.5% (24) of the protocols available in our Nature Protocols Exchange collection reported the provenance (Fig. 2). None of the protocols collected from Cold Spring Harbor Protocols or Bio-protocols had been updated–last checked December 2017.

As a result of the workshops, domain experts exposed the importance of including these three data elements in our checklist. For instance, readers sometimes need to contact the authors to ask about specific information (quantity of the sample used, the storage conditions of a solution prepared in the lab, etc.); occasionally, the correspondent author does not respond because he/she has changed his/her email address, and searching for the full name could retrieve multiple results. By using author IDs, this situation could be resolved. The experts asserted that well-documented provenance helps them to know where the protocol comes from and whether it has changed. For example, domain experts expressed their interest in knowing where a particular protocol was published for the first time, who has reused it, how many research papers have used it, how many people have modified it, etc. In a similar way, domain experts also expressed the need for a version control system that could help them to know and understand how, where and why the protocol has changed. For example, researchers are interested in tracking changes in quantities, reagents, instruments, hints, etc. For a complete description of the bibliographic data elements proposed in our checklist, see below.

Title. The title should be informative, explicit, and concise (50 words or fewer). The use of ambiguous terminology and trivial adjectives or adverbs (e.g., novel, rapid, efficient, inexpensive, or their synonyms) should be avoided. The use of numerical values, abbreviations, acronyms, and trademarked or copyrighted product names is discouraged. This definition was adapted from BioTechniques (Giraldo, Garcia & Corcho, 2018b). In Table 8, we present examples illustrating how to define the title.

Author name and author identifier. The full name(s) of the author(s) is required together with an author ID, e.g., ORCID (ORCID, 2017) or research ID (ResearcherID, 2017). The role of each author is also required; depending on the domain, there may be several roles. It is important to use a simple word that describes who did what. Publishers, laboratories, and authors should enforce the use of an “author contribution section” to identify the role of each author. We have identified two roles that are common across our corpus of documents.

• Creator of the protocol: This is the person or team responsible for the development or adaptation of a protocol.

• Laboratory-validation scientist: Protocols should be validated in order to certify that the processes are clearly described; it must be possible for others to follow the described processes. If applicable, statistical validation should also be addressed. The validation may be procedural (related to the process) or statistical (related to the statistics). According to the Food and Drug Administration (FDA) (FDA, 2017), validation is “establishing documented evidence which provides a high degree of assurance that a specific process will consistently produce a product meeting its predetermined specifications and quality attributes” (Das, 2011).

Updating the protocol. The peer-reviewed and non peer-reviewed repositories of protocols should encourage authors to submit updated versions of their protocols; these may be corrections, retractions, or other revisions. Extensive modifications to existing protocols could be published as adapted versions and should be linked to the original protocol. We recommended to promote the use of a version control system; in this paper we suggest to use the version control guidelines proposed by the National Institute of Health (NIH) (NIH, 2017).

• Document dates: Suitable for unpublished protocols. The date indicating when the protocol was generated should be in the first page and, whenever possible, incorporated into the header or footer of each page in the document.

• Version numbers: Suitable for unpublished protocols. The current version number of the protocol is identified in the first page and, when possible, incorporated into the header or footer of each page of the document.

  • Draft document version number: Suitable for unpublished protocols. The first draft of a document will be Version 0.1. Subsequent drafts will have an increase of “0.1” in the version number, e.g., 0.2, 0.3, 0.4, ... 0.9, 0.10, 0.11.

  • Final document version number and date: Suitable for unpublished and published protocols. The author (or investigator) will deem a protocol final after all reviewers have provided final comments and these have been addressed. The first final version of a document will be Version 1.0; the date when the document becomes final should also be included. Subsequent final documents will have an increase of “1.0” in the version number (1.0, 2.0, etc.).

• Documenting substantive changes: Suitable for unpublished and published protocols. A list of changes from the previous drafts or final documents will be kept. The list will be cumulative and identify the changes from the preceding document versions so that the evolution of the document can be seen. The list of changes and consent/assent documents should be kept with the final protocol.

Provenance of the protocol. The provenance is used to indicate whether or not the protocol results from modifying a previous one. The provenance also indicates whether the protocol comes from a repository, e.g., Nature Protocols Exchange, protocols.io (Teytelman et al., 2016), or a journal like JoVE, MethodsX, or Bio-Protocols. The former refers to adaptations of the protocol. The latter indicates where the protocol comes from. See Table 9.

License of the protocol. The protocols should include a license. Whether as part of a publication or, just as an internal document, researchers share, adapt and reuse protocols. The terms of the license should facilitate and make clear the legal framework for these activities.

Data elements of the discourse

Here, we present the elements considered necessary to understand the suitability of a protocol. They are the “overall objective or purpose”, “applications”, “advantages,” and “limitations”. 100% of the analyzed guidelines for author suggest the inclusion of these four elements in the abstract or introduction section. However, one or more of these four elements were not reported. For example, “limitations” was reported in only 20% of the protocols from Genetic and Molecular Research and PLOS One, and in 40% of the protocols from Springer. See Fig. 3.

Interestingly, 83% of the respondents considered the “limitations” to be a data element that is necessary when reporting a protocol. In the last meeting, participants considered that “limitations” represents an opportunity to make suggestions for further improvements. Another data element discussed was “advantages”; 43% of the respondents considered the “advantages” as a data element that is necessary to be reported in a protocol. In the last meeting, all participants agreed that “advantages” (where applicable) could help us to compare a protocol with other alternatives commonly used to achieve the same result. For a complete description of the discourse data elements proposed in our checklist, see below.

Overall objective or Purpose. The description of the objective should make it possible for readers to decide on the suitability of the protocol for their experimental problem. See Table 10.

Application of the protocol. This information should indicate the range of techniques where the protocol could be applied. See Table 10.

Advantage(s) of the protocol. Here, the advantages of a protocol compared to other alternatives should be discussed. See Table 10. Where applicable, references should be made to alternative methods that are commonly used to achieve the same result.

Limitation(s) of the protocol. This datum includes a discussion of the limitations of the protocol. This should also indicate the situations in which the protocol could be unreliable or unsuccessful. See Table 10.

Data elements for materials

From the analyzed guidelines for authors, the datum “sample description” was considered only in the Current Protocols guidelines. The “laboratory consumables or supplies” datum was not included in any of the analyzed guidelines. See Fig. 4.

Our Current Protocols collection includes documents about toxicology, microbiology, magnetic resonance imaging, cytometry, chemistry, cell biology, human genetics, neuroscience, immunology, pharmacology, protein, and biochemistry; for these protocols the input is a biological or biochemical sample. This collection also includes protocols in bioinformatics with data as the input. 100% of the protocols from our Current Protocols collection includes information about the input of the protocol (biological/biochemical sample or data). In addition, 87% of protocols from this collection include a list of materials or resources (reagents, equipment, consumables, software, etc.).

We also analyzed the protocols from our MethodsX collection. We found that despite the exclusion of the sample description in guidelines for authors, the authors included this information in their protocols. Unfortunately, these protocols do not include a list of materials. Only 29% of the protocols reported a partial list of materials. For example, the protocol published by Vingataramin & Frost (2015), includes a list of recommended equipment but does not list any of the reagents, consumables, or other resources mentioned in the protocol instructions. See Fig. 5.

Domain experts considered that the input of the protocol (biological/biochemical sample or data) needs an accurate description; the granularity of the description varies depending on the domain. If such description is not available then the reproducibility could be affected. In addition, domain experts strongly suggested to include consumables in the checklist. It was a general surprise not to find these data elements in the guidelines for authors that we analyzed. Domain experts shared with us bad experiences caused by the lack of information about the type of consumables. Some of the incidents that may arise from the lack of this information include: (i) cross contamination, when no information suggesting the use of filtered pipet tips is available; (ii) misuse of containers, when no information about the use of containers resistant to extreme temperatures and/or impacts is available; (iii) misuse of containers, when a container made of a specific material should be used, e.g., glass vs. plastic vs. metal. This is critical information; researchers need to know if reagents or solutions prepared in the laboratory require some specific type of containers in order to avoid unnecessary reactions altering the result of the assay. Presented below is the set of data elements related to materials or resources used for carrying out the execution of a protocol.

Sample. This is the role played by a biological substance; the sample is an experimental input to a protocol. The information required depends on the type of sample being described and the requirements from different communities. Here, we present the data elements for samples commonly used across the protocols and guidelines that we analyzed.

• Bio-source properties:

  • Strain, genotype or line: This datum is about subspecies such as ecotype, cultivar, accession, or line. In the case of crosses or breeding results, pedigree information should also be provided.

  • Starting material: This datum is about the physical biological specimen from which your experimental data are derived. The starting material could be a whole organism, or a part of this.

    • whole organism Typical examples are multicellular animals, plants, and fungi; or unicellular microorganisms such as a protists, bacteria, and archaea.

    • organism part Typical examples of an organism part include a cell line, a tissue, an organ, corporal bodily fluids protoplasts, nucleic acids, proteins, etc.

    • organism/sample identifier This is the unique identifier assigned to an organism. The NCBI taxonomy id, also known as “taxid”, is commonly used to identify an organism; the Taxonomy Database is a curated classification and nomenclature for all organisms in the public sequence databases. Public identification systems, e.g., the Taxonomy Database, should be used when ever possible. Identifiers may be internal; for instance, laboratories often have their own coding system for generating identifiers. When reporting internal identifiers it is important to also state the source and the nature (private or pubic) of the identifier, e.g., A0928873874, barcode (CIAT-DAPA internal identifier) of a specimen or sample.

  • Amount of Bio-Source: This datum is about mass (mg fresh weight or mg dry weight), number of cells, or other measurable bulk numbers (e.g., protein content).

  • Developmental stage: This datum includes age and gender (if applicable) of the organism.

  • Bio-source Supplier: This datum is defined as a person, company, laboratory or entity that offers a variety of biosamples or biospecimens.

• Growth conditions:

  • Growth substrates: This datum refers to an hydroponic system (type, supplier, nutrients, concentrations), soil (type, supplier), agar (type, supplier), and cell culture (media, volume, cell number per volume).

  • Growth environment: This datum includes, but is not limited to, controlled environments such as greenhouse (details on accuracy of control of light, humidity, and temperature), housing conditions (light/dark cycle), and non-controlled environments such as the location of the field trial.

  • Growth time: This datum refers to the growth time of the sample prior to the treatment.

• Sample pre-treatment or sample preparation: This datum refers to collection, transport, storage, preparation (e.g., drying, sieving, grinding, etc.), and preservation of the sample.

Laboratory equipment. The laboratory equipment includes apparatus and instruments that are used in diagnostic, surgical, therapeutic, and experimental procedures. In this subsection, all necessary equipment should be listed; manufacturer name or vendor (including the homepage), catalog number (or model), and configuration of the equipment should be part of this data element. See Table 11.

• Laboratory equipment name: This datum refers to the name of the equipment as it is given by the manufacturer (e.g., FocalCheck fluorescence microscope test slide).

• Manufacturer name: This datum is defined as a person, company, or entity that produces finished goods (e.g., Life Technologies, Zeiss).

• Laboratory equipment ID (model or catalog number): This datum refers to an identifier provided by the manufacturer or vendor (e.g., F36909—catalog number for FocalCheck fluorescence microscope test slide from Life Technologies).

• Equipment configuration: This datum should explain the configuration of the equipment and the parameters that make it possible to carry out an operation, procedure, or task (e.g., the configuration of an inverted confocal microscope).

Laboratory consumables or supplies. The laboratory consumables include, amongst others, disposable pipettes, beakers, funnels, test tubes for accurate and precise measurement, disposable gloves, and face masks for safety in the laboratory. In this subsection, a list with all the consumables necessary to carry out the protocol should be presented with manufacturer name (including the homepage) and catalog number. See Table 12.

• Laboratory consumable name: This datum refers to the name of the laboratory consumable as it is given by the manufacturer e.g., Cryogenic Tube, sterile, 1.2 ml.

• Manufacturer name: This datum is defined as a person, enterprise, or entity that produces finished goods (e.g., Nalgene, Thermo-scientific, Eppendorf, Falcon)

• Laboratory consumable ID (catalog number): This datum refers to an identifier provided by the manufacturer or vendor; for instance, 5000-0012 (catalog number for Cryogenic Tube, sterile, 1.2 mL from Nalgene).

Recipe for solutions. A recipe for solutions is a set of instructions for preparing a particular solution, media, buffer, etc. The recipe for solutions should include the list of all necessary ingredients (chemical compounds, substance, etc.), initial and final concentrations, pH, storage conditions, cautions, and hints. Ready-to-use reagents do not need to be listed in this category; all purchased reagents that require modification (e.g., a dilution or addition of β-mercaptoethanol) should be listed. See Table 13 for more information.

• Solution name: This is the name of the preparation that has at least 2 chemical substances, one of them playing the role of solvent and the other playing the role of solute. If applicable, the name should include the following information: concentration of the solution, final volume and final pH. For instance, Ammonium bicarbonate (NH4HCO3), 50 mM, 10 ml, pH 7.8.

• Chemical compound name or reagent name: This is the name of a drug, solvent, chemical, etc.; for instance, agarose, dimethyl sulfoxide (DMSO), phenol, sodium hydroxide. If applicable, a measurable property, e.g., concentration, should be included.

• Initial concentration of a chemical compound: This is the first measured concentration of a compound in a substance.

• Final concentration of chemical compound: This is the last measured concentration of a compound in a substance.

• Storage conditions: This datum includes, among others, shelf life (maximum storage time) and storage temperature for the solutions e.g., “Store the solution at room temperature”, “maximum storage time, 6 months”. Specify whether or not the solutions must be prepared fresh.

• Cautions: Toxic or harmful chemical compounds should be identified by the word ‘CAUTION’ followed by a brief explanation of the hazard and the precautions that should be taken when handling e.g., “CAUTION: NaOH is a very strong base. Can seriously burn skin and eyes. Wear protective clothing when handling. Make in fume hood”.

• Hints: The “hints” are commentaries or “tips” that help the researcher to correctly prepare the recipe e.g., “Add NaOH to water to avoid splashing”.

Reagents. A reagent is a substance used in a chemical reaction to detect, measure, examine, or produce other substances. List all the reagents used when performing the protocol, the vendor name (including homepage), and catalog number. Reagents that are purchased ready-to-use should be listed in this section. See Table 14.

• Reagent name: This datum refers to the name of the reagent or chemical compound. For instance, “Taq DNA Polymerase from Thermus aquaticus with 10X reaction buffer without MgCl2”.

• Reagent vendor or manufacturer: This is the person, enterprise, or entity that produces chemical reagents e.g., Sigma-Aldrich.

• Reagent ID (catalog number): This is an identifier provided by the manufacturer or vendor. For instance, D4545-250UN (catalog number for Taq DNA Polymerase from Thermus aquaticus with 10X reaction buffer without MgCl2 from Sigma-Aldrich).

Kits. A kit is a gear consisting of a set of articles or tools for a specific purpose. List all the kits used when carrying out the protocol, the vendor name (including homepage), and catalog number.

• Kit name: This datum refers to the name of the kit as it is given by the manufacturer e.g., Spectrum Plant Total RNA Kit, sufficient for 50 purifications.

• Kit vendor or manufacturer: This is the person, enterprise, or entity that produces the kit e.g., Sigma-Aldrich.

• Kit ID (catalog number): This is an identifier provided by the manufacturer or vendor e.g., STRN50, catalog number for Spectrum^™ Plant Total RNA Kit, sufficient for 50 purifications.

Software. Software is composed of a series of instructions that can be interpreted or directly executed by a processing unit. In this subsection, please list software used in the experiment including the version, as well as where to obtain it.

• Software name: This datum refers to the name of the software. For instance, “LightCycler 480 Software”.

• Software version: A software version number is an attribute that represents the version of software e.g., Version 1.5.

• Software availability: This datum should indicate where the software can be downloaded from. If possible, license information should also be included; for instance, https://github.com/MRCIE-U/ariesmqtl, GPL3.0.

Data elements for the procedure

All the analyzed guidelines include recommendations about how to document the instructions; for example, list the steps in numerical order, use active tense, organize the procedures in major stages, etc. However, information about documentation of alternative, optional, or parallel steps (where applicable) and alert messages such as critical steps, pause point, and execution time was infrequent (available in less than 40% of the guidelines). See Fig. 6.

We chose a subset of protocols (12 from our Plant Methods collection, 7 from our Biotechniques collection, and five unpublished protocols from CIAT) to review which data elements about the procedure were documented. 100% of the protocols have steps organized in major stages. 100% of the unpublished protocols list the steps in numerical order, and nearly 60% of the protocols from Plant Methods and Biotechniques followed this recommendation. Alert messages were included in 67% of the Plant Methods protocols and in 14% of the Biotechniques protocols. Neither of the five unpublished protocols included alert messages. Troubleshooting was reported in just a few protocols; this datum was available in 8% of the Plant Methods protocols and in 14% of the Biotechniques protocols. See Fig. 7.

In this stage, the discussion with domain experts started with the description of steps. In some protocols, the steps are poorly described; for instance, some of them include working temperatures, e.g., cold room, on ice, room temperature; but, what exactly do they mean? Steps involving centrifugation, incubation, washing, etc., should specify conditions, e.g., time, temperature, speed (rpm or g), number of washes, etc. For experts, alert messages and troubleshooting (where applicable) complement the description of steps and facilitate a correct execution. This opinion coincides with the results of the survey, where troubleshooting and alert messages such as critical steps, pause points, and timing were considered relevant by 83%–87% of the respondents. The set of data elements related to the procedure is presented below.

• Recommendation 1. Whenever possible, list the steps in numerical order; use active tense. For example: “Pipette 20 ml of buffer A into the flask,” as opposed to “20 ml of buffer A are/were pipetted into the flask” (Nature Protocols, 2012).

• Recommendation 2. Whenever there are two or more alternatives, these should be numbered as sets of consecutive steps Wiley’s Current Protocols (2012). For example: “Choose procedure A (steps 1–10) or procedure B (steps 11–20); then continue with step 21 . . .”. Optional steps or steps to be executed in parallel should also be included.

• Recommendation 3. For techniques comprising a number of individual procedures, organize these in the exact order in which they should be executed (Nature Protocols, 2012).

• Recommendation 4. Description of steps. Those steps that include working temperatures, e.g., cold room, on ice, room temperature, should be clearly specified. From the European Pharmacopoeia (Pharm.Eur.) (ECA Foundation, 2017), World Health Organization resource guidance (WHO guidance) (WHO, 2003), and the U.S. Pharmacopeia (USP) (USP, 2018), the most common storage conditions were extracted, see below:

  • Frozen/deep-freeze temperature (−20 °C to −15 °C)

  • Refrigerator, cold room or cold temperature (2 °C to 8 °C)

  • Cool temperature (8 °C to 15 °C)

  • Room/Ambient temperature (15 °C to 25 °C)

  • Warm/Lukewarm temperature (30 °C to 40 °C)

  For centrifugation steps, specify time, temperature, and speed (rpm or g). Always state whether to discard/keep the supernatant/pellet. For incubations, specify time, temperature, and type of incubator. For washes, specify conditions e.g., temperature, washing solution and volume, specific number of washes, etc.

Useful auxiliary information should be included in the form of “alert messages”. The goal is to remind or alert the user of a protocol with respect to issues that may arise when executing a step. These messages may cover special tips or hints for performing a step successfully, alternate ways to perform the step, warnings regarding hazardous materials or other safety conditions, time considerations. For instance, pause points, speed at which the step must be performed and storage information (temperature, maximum duration) (Wiley’s Current Protocols, 2012).

• Critical steps: Highlight critical steps in the protocol and give indications that help to carry these out in a precise manner. For instance, time and temperature information if these are deem crucial. Or, whether the use of RNase free solutions is required. Information should be provided in order to indicate how these steps are critical and how to overcome the issues. “Critical Steps” should help the user to maximize the likelihood of success; use the heading CRITICAL STEP followed by a brief explanation. See Table 15.

• Pause point: This datum is appropriate after steps in the protocol where the procedure can be stopped. i.e., when the experiment can be stopped and resumed at a later point in time. Any PAUSE POINTS should be indicated with a brief description of the options available. See Table 15.

• Timing: This datum is used to include the approximate time of execution of a step or set of steps. Timing could also be indicated at the beginning of the protocol. See Table 15.

• Hints: Provide any commentary, note, or hints that will help the researcher to correctly perform the protocol. See Table 15.

• Troubleshooting: This datum is used to list common problems, possible causes, and solutions/methods of correction. This can be submitted as a 3-column table or listed in the text. An example is presented in “Table 1.Troubleshooting table”, available at Rohland & Hofreiter (2007).