Tolerogenic dendritic cell reporting: Has a minimum information model made a difference?

Minimum information models are reporting frameworks that describe the essential information that needs to be provided in a publication, so that the work can be repeated or compared to other work. In 2016, Minimum Information about Tolerogenic Antigen-Presenting cells (MITAP) was created to standardize the reporting on tolerogenic antigen-presenting cells, including tolerogenic dendritic cells (tolDCs). tolDCs is a generic term for dendritic cells that have the ability to (re-)establish immune tolerance; they have been developed as a cell therapy for autoimmune diseases or for the prevention of transplant rejection. Because protocols to generate these therapeutic cells vary widely, MITAP was deemed to be a pivotal reporting tool by and for the tolDC community. In this paper, we explored the impact that MITAP has had on the tolDC field. We did this by examining a subset of the available literature on tolDCs. Our analysis shows that MITAP is used in only the minority of relevant papers (14%), but where it is used the amount of metadata available is slightly increased over where it is not. From this, we conclude that MITAP has been a partial success, but that much more needs to be done if standardized reporting is to become common within the discipline.


INTRODUCTION
A large amount of background information is required to fully understand the context, methods, data and conclusions that pertain to an experiment. Minimum information models (MIMs) provide a framework for reporting all the essential information (metadata) about experimental work and they have become popular among the scientific community. They are designed to make research more reproducible; following their guidelines ensures that the minimum necessary information about the experimental work is presented in a report or publication, so that other researchers can re-use or re-purpose the data and methods.
From 2014 to 2016, a group of researchers working in the field of tolerogenic dendritic cells (tolDCs) came together to generate a MIM for their field. tolDCs are antigen-presenting cells that induce or restore immune tolerance (Fucikova et al., 2019). These cells show great promise as a therapeutic tool for the treatment of conditions caused by a breach in immune tolerance (e.g., autoimmune diseases) or for the prevention of graft-rejection. However, the diverse range of experimental designs and reagents used to generate tolDCs precluded meaningful comparisons between different tolDC types. It was anticipated that a MIM would improve the transparency, reproducibility and data interpretation of the different tolDC types. MITAP (Minimum Information about Tolerogenic Antigen-Presenting cells) was published in 2016 (Lord et al., 2016); the more general name was chosen to allow for the inclusion of other tolerogenic antigen-presenting cell types, for example regulatory macrophages (MRegs).
In total MITAP took 18 months to create as a collaboration by experts in the tolDC field. The whole procedure involved a series of interactive workshops in which several exercises were conducted to gather the relevant basic vocabulary used within the community; to obtain feedback on the draft reporting guidelines, and finally to test its comprehensibility by the end-users. The full MITAP document, including the checklist, can be found on https://w3id.org/ontolink/mitap. MITAP consists of 4 sections that were considered to be essential by the experts for reporting all necessary information about the generation of tolDCs and other tolerogenic antigen-presenting cell (tolAPC) products. The most crucial stages of the production process are encapsulated in these sections in an orderly manner. They are summarized below: Section 1. Cells before: This section describes characteristics of the cells before they undergo any manipulation such as (a) essential information about the donor, (b) source of the cells, (c) the cell extraction method (d), cell phenotype after extraction and (e) cell number and viability.
Section 2. Differentiation and induction of tolerogenicity: This section describes the protocol that has been used to differentiate and/or induce tolerogenicity in the cells described in section 1. There are five subsections giving details on (a) preculture conditions, (b) culture conditions, (c) the protocol used to induce differentiation and tolerogenicity of the cells, (d) loading of cells antigen, and (e) the way cells are stored immediately after culture.
Section 3. Cells after: This section describes the characteristics and state of the cells after the differentiation/induction of tolerogenicity process has taken place. This section has three subsections, two of which provide similar basic details as in section 1 on the (a) cell phenotype and (c) cell number and viability. Another part (b) focuses on functional in vitro assays such as migratory capacity of the cells, or their ability to induce T regulatory cells.
Section 4. About the protocol: The final section of MITAP describes the general features of the protocol as a whole, such as (a) any external regulatory authorities or guidelines followed; (b) purpose of the cells; (c) whether cell product is applied in an autologous, allogeneic, xenogeneic or syngeneic manner, and finally (d) contact details of the authors.
Here, we examined the usage of MITAP in the tolDC community. More specifically, we addressed the question of whether these reporting guidelines improved the provision of relevant metadata in papers published after the MITAP publication date.

How many papers have used MITAP?
The first stage of our analysis was to ask the question of how many papers used MITAP. To answer this, we assumed that any paper that directly used MITAP would include it in the reference list. Therefore, we used Google scholar and downloaded all the papers that had cited MITAP; we found that MITAP had 40 citations in January 2021. By inspection, we found that not all of these papers used MITAP; some papers, for example reviews, referred to MITAP as an example of a MIM. In fact, only 10 out of the 40 papers that had cited MITAP were using it directly as a reporting framework.

How many papers could have used MITAP?
While it is useful to know the actual penetrance of MITAP, we also wished to understand how many papers that are reporting results about tolDCs could have used MITAP but did not. This question is harder to answer exactly, but we calculated an estimate. Our overall approach is depicted in Fig. 1. We assumed that most of the papers related to tolDCs are available on PubMed. We therefore searched PubMed with a variety of keywords appropriate for papers related to the MITAP subject matter, restricting the search to papers that were published after MITAP itself was published.
Of course, to use the number of search results as an estimate for the total number of papers, we must be sure of our query terms. We achieved this by testing the performance of each query by calculating the precision (how accurate the results were) and recall (how complete the results were). We could test for recall because we had a set of papers that directly cited MITAP; these therefore fall into the category of papers that could (and in this case did) use MITAP. If our query has 100% recall, therefore, all of these papers (and others) should be retrieved. We calculated the precision by simply reading a subset of 20 papers from each query and making a expert judgement about whether they were relevant.
To provide a base line for our queries, we initially generated keywords using inverse document frequency (IDF) from the titles and abstracts of 10 relevant MITAP citations. All the queries were refined by putting a 'date of publication' filter, so that only papers published after MITAP were retrieved, since no papers published before MITAP could have used it. In addition, the review papers were also excluded by applying a ''NOT'' filter on the query, since these were not relevant to this investigation. A query using only keywords that were given in the original MITAP paper retrieved a smaller number of papers, but the recall was also lower (Table 1). Finally, we modified these queries based on expert judgement of the authors resulting in the manual adjusted queries. The precision and recall were higher, but the number of retrieved papers was large. Therefore, the query was further refined by adding keywords such as ''generate'', ''produce'' or ''induce'' to get only those papers that reported on tolDCs that were produced as part of an experiment. The precision and recall of manual adjustment-2 retrieved results were found appropriate among other queries. Thus, this list of 72 papers was decided to be the total number of tolAPC papers published after MITAP (of which the majority focused on tolDCs, as expected). Hence only (10/72) 14% of researchers have used MITAP so far to report their data, while the remaining 62 tolAPC papers did not use MITAP for data reporting. Figure 1 The flowchart describing the overall approach of this study. The above diagram represents the overall approach of this paper. The process begins with specific SQL query based search on PubMed, with further refining the search by using most relevant keywords, a corpus of all the papers published on generating tolDCs is established. An unbiased random selection of 10 papers is performed from each category and these papers are further investigated by experts in the field of tolDCs.

Comparison of papers with and without MITAP
The purpose of the MITAP document is to ensure that authors provide sufficient basic information about the generation of tolDCs or other types of tolAPC. As shown in the original MITAP paper, complete information is not being provided routinely. 5 years after the MITAP publication, we wanted to re-check the status of information provided by the tolDC community. We therefore applied the MITAP checklist to 10 papers that used MITAP and 10 papers that could have used MITAP but did not. These papers are listed in the Further Reading section, but the sequence in Fig. 2 is different and anonymized. Red sections in the heatmap represent the information missing from the papers, green areas show the information directly provided in the paper and yellow areas show that the information is partially available (e.g., information is available in a referenced paper but  not in the paper itself), or not relevant to the paper. Figure 2 shows that not all the sections of the MITAP checklist were completed by both category of papers. We therefore carried out further analysis to check the percentage of reported parameters for each section and performed statistical analysis. As the data is categorical and does not follow any distribution, the non-parametric McNemar test was carried out, using python 3.9 mlxtend library. For both MITAP and non-MITAP papers, Section 1 is the least reported section with less than 40% of the parameters reported, whereas for all other sections >50% of the parameters were reported. MITAP papers performed marginally better than non-MITAP papers for sections 1, 2 and 4; there were no significant differences for section 3 ( Fig. 3 and Table 2).

Comparison of MITAP with other MIMs
While it is known that high quality data reporting is important and essential for reproducible research, as we have shown, MITAP is currently used by only 14% of tolDC publications. There has been much recent interest on the lack of reproducibility in many areas of biology and immunology, so we doubt that these figures for the tolDC community are unusual. This may be because following standards for the design and subsequent reporting of experiments and subsequent analysis is complex or time consuming. There are a number of ways to reduce the effort of this data reporting; of these, MIMs are seen as an effective mechanism for standardisation, as evidenced by the number of them that being introduced into the medical field (Fuchs et al., 2018). Figure 4 shows an increasing trend of MIMs in the medical field in the last 20 years. Because MITAP was used by a smaller number of published papers than we might have hoped, we analysed and compared usage of other MIMs to MITAP. MITAP was published   (Brazma et al., 2001;Field et al., 2008). As dendritic cells are a type of immune cells, we looked at other immunology related MIMs such as MIATA (Minimum Information about T cell Assays) and MITREG (Minimum Information about T Regulatory Cells) (Britten et al., 2011;Fuchs et al., 2018). Figure 6 shows the citation statistics of these five MIMs compared to MITAP. MIGS and MIAME are MIMs for ''omics'' data and are both broadly applied in many areas of biology, so might be expected to be widely used, but it can be clearly seen that the citations are still limited. Similarly, MIATA and T cells also cover substantially bigger and older research fields than the tolDC field but show maximum 16 citations only in the peak year.
These citation statistics suggest that MITAP is performing well and is a sufficient tool to implement standardisation in the field of dendritic cells. Even when the MITAP checklist  is not followed completely, it still fulfils the requirement for providing a minimum set of most important entities for tolDCs, as well as increasing the data reporting.

DISCUSSION
Over 5 years ago, investigators in the field of tolDC came together to create MITAP, a minimum guidelines tool for reporting of the protocols to generate these cells. By using MITAP on extant papers that were published before MITAP came into existence, we showed that a large proportion of papers lacked sufficient data required to interpret and reproduce the generation of these tolerogenic cells (Lord et al., 2016). Here, we investigated usage of MITAP, and whether it improved the description of the four protocol sections: (1) Cells before; (2) differentiation and induction of tolerogenicity; (3) cells after, and (4) about the protocol. Despite the highly collaborative nature of building MITAP, which involved many experts in the field, we have found that the use of MITAP is surprisingly low: only 14% of research papers that could have used MITAP, actually did. Although encouragingly, those papers that used MITAP performed slightly better on reporting relevant data across three of the four protocol sections, experimental details remained under-reported especially in section 1 ('Cells before') and 2 ('Induction').
There were no significant differences in reporting on section 3 ('Cells after') between MITAP-and non-MITAP papers. This can most likely be explained by the fact that section 3 deals with the final tolDC product itself, as these cells are the main focus of these papers, details on these cells are usually well reported. In contrast, section 1 and 2, which describe the source and features of the cells before (section 1) they undergo the differentiation process to become tolDCs (section 2), were significantly under-reported in non-MITAP papers compared to MITAP papers. It is not known whether this is because the authors either did not have the relevant information or data available or just did not report it. Nevertheless, under-reporting of these sections will ultimately make it more difficult for investigators to reproduce published data and/or to make comparisons between different types of tolDCs. As it is becoming increasingly clear that considerable heterogeneity exists between different tolDC products, but also between tolDC products derived from different patients' groups (Navarro- Barriuso, Mansilla & Martínez-Cáceres, 2018;Navarro-Barriuso et al., 2019), it is becoming even more important to improve reporting on the cell source and culture protocol to generate tolDCs.
There is a growing recognition that open data reporting and standardisation are necessary for repurposing or reusing data to discover valuable insights from past work, thus promoting research transparency. In this context, the 14% usage of MITAP might seem low or disappointing. However, our experience shows that the take up of MIMs takes time, despite the general appreciation of their importance. A possible reason for this could be the disparity between effort and reward: for the person producing the data, standardisation may be time-consuming with low reward; conversely, for those consuming the data, they provide high reward, maximize repurposing and reuse, with much less effort that otherwise; a MIM can provide standardisation and tooling can help to reduce the effort for data producers, but it is a high effort activity for those experts developing the MIMs. We also note that, from an informal analysis of the papers, that MITAP usage seems higher among papers coming from European authors; this perhaps suggests that more outreach work is needed, with scientists in other parts of the world.
Perhaps the main way to increase the penetrance of a MIM is the availability of a relevant data repository. MIAME is one of the most successful MIMs and has relevant repositories such as ArrayExpress or GEO requiring MIAME compliant data submission. As a result, many journals indirectly also require researchers to follow MIAME to submit the data. Other factors to the success of MIM can be the availability of software to record the metadata combined with a database to retrieve the data along with the metadata efficiently.
Our experiments demonstrate that MITAP is sufficient for the reporting of data; as a result, it has fulfilled its key objective. However, to make its use more widespread in the field, we need to introduce an online platform where researchers can record, edit, and save the metadata in a user-friendly manner.

ADDITIONAL INFORMATION AND DECLARATIONS Funding
This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 860003 and BBSRC/GSK Case studentship award BB/S507039/1. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
• Megan Paterson performed the experiments, authored or reviewed drafts of the article, and approved the final draft.
• Catharien M.U. Hilkens conceived and designed the experiments, authored or reviewed drafts of the article, and approved the final draft.
• Phillip Lord conceived and designed the experiments, authored or reviewed drafts of the article, and approved the final draft.

Data Availability
The following information was supplied regarding data availability: Our study does not produce or use any raw data

Supplemental Information
Supplemental information for this article can be found online at http://dx.doi.org/10.7717/ peerj.15352#supplemental-information.