Panoramic quality assessment tool for investigator initiated trials

Objectives Quality can be a challenge for Investigator initiated trials (IITs) since these trials are scarcely overseen by a sponsor or monitoring team. Therefore, quality assessment for departments managing clinical research grants program is important and urgently needed. Our study aims at developing a handy quality assessment tool for IITs that can be applied by both departments and project teams. Methods The framework of the quality assessment tool was developed based on the literature studies, accepted guidelines and the Delphi method. A total of 272 ongoing IITs funded by Shanghai non-profit organizations in 2015 and 2016 were used to extract quality indexes. Confirmatory factor analysis (CFA) was used to further evaluate the validity and feasibility of the conceptual quality assessment tool. Results The tool consisted of 4 critical quality attributes, including progress, quality, regulation, scientificity, and 13 observed quality indexes. A total of 257 IITs were included in the validity and feasibility assessment. The majority (60.29%) were Randomized Controlled Trial (RCT), and 41.18% were multi-center studies. In order to test the validity and feasibility of IITs quality assessment tool, CFA showed that the model fit the data adequately. (CMIN/DF = 1.868, GFI = 0.916; CFI = 0.936; TLI = 0.919; RMSEA = 0.063; SRMR = 0.076). Different types of clinical studies fit well in the tool. However, RCT scored lower than prospective cohort and retrospective study in enrollment progress (7.02 vs. 7.43, 9.63, respectively). Conclusion This study established a panoramic quality assessment tool based on the Delphi method and CFA, and the feasibility and effectiveness of the tool were verified through clinical research examples. The use of this tool can help project management departments effectively and dynamically manage research projects, rationally allocate resources, and ensure the quality of IITs.


Introduction
Investigator initiated trials (IITs) complement the industrysponsored trials (ISTs), optimize existing therapies or treatment approaches and attempt to answer clinical problems (1,2). In addition, IITs do not only facilitate a better understanding of disease domain and drug effect, but help translate academic research into clinical practice. Recent literature also reported that IITs have changed the practice of medicine (3).
Industry-sponsored trials have rigorous monitoring and auditing to ensure the authenticity and reliability of data, while IITs often lack resources and may not have similar quality checks (4,5). IITs are equally important to ISTs as they explore the use of marketed drugs for new indications and clinical diagnosis or treatment effect comparison (4). Thus the quality of IITs should be taken seriously (6). For hospitals, an alternative quality management system could be adopted to alleviate regulatory pressures (7), and a comprehensive and feasible quality assessment tool is also urgently required as IITs faced challenges both in design and conduct.
Over the past decade, more and more studies emphasized on the importance of methodology and the quality of research report, both in IITs and ISTs (8,9). Several publications highlighted that risk-adapted monitoring was important for quality control and sufficient to identify critical questions in the conduct of clinical trials (10,11). Quality assessment tools for research design or reporting were abundant, while few for operation and funding decision (12). The Risk-Based Monitoring Toolbox of European Clinical Research Infrastructure Network provided information on tools available for risk assessment, monitoring and study conduct. The toolbox was mainly created following literature review or surveys, and was a collection of risk-based tools and strategies (13). Take RACT and ADAMON as examples. In 2013, TransCelerate BioPharma, an independent non-profit organization, developed a Risk Assessment and Categorization Tool (RACT), which many biopharmaceutical companies have used to assess the risk level of clinical trials before the start of the trial as well as regular checkups. Despite that the RACT offered a very useful methodology for risk assessments of study level, it had some weaknesses when used to evaluate IITs. The RACT was prone to subjectivity and lacked important categories (14). Adapters Monitoring (ADAMON) risk scale can be used for IITs, but it's just a tool for assessing the required amount of on-site monitoring, not whole quality. Furthermore, Patwardhan et al. https://www.tmf-ev.de/ADAMON_EN/Projectdescription.aspx (15) drew up a checklist consisting of various criteria that were essential in IITs documentation, while not suitable to assess large-scale IITs. Systematic study about quality assessment tools designed for large-scale IITs was rare. The main goal of this study is to develop an operational quality assessment tool for IITs to enhance the quality of clinical research, ensure the safety of subjects and the authenticity and reliability of data, and avoid waste of resources.

Index system construction
The quality indicators were extracted based on the following three principles. The indicators can be (1) used to evaluate the quality of clinical research (2) relatively simple and easy to understand, and (3)  Investigator initiated trials should follow GCP principles to ensure protection of the trial subjects and assures quality and credibility of the data obtained. We used focus group interviews to collect and build critical attributes of IITs that the funding agencies like NIH, clinical research organizations and hospitals is most concerned about. Finally, four critical attributes of IITs were obtained, namely: progress, quality, regulation, and scientificity. We further expanded indices according to four critical attributes of IIT by referring to literature and accepted guidelines ( Table 1). The importance and feasibility of the indicators were further evaluated through two rounds of the Delphi method. We consulted more than 20 experts with senior titles in each round, including clinical research methodologists, research managers, and clinicians engaged in clinical research. Furthermore, this research used confirmatory factor analysis (CFA) and structural equation modeling as tools to evaluate the structural validity of the index system ( Figure 1).  properly sample size estimation 2 , safety endpoints.

Yes No
Can't answer Not applicable Study protocol dissemination Study protocol published, study protocol provided through internet.

Yes No
Can't answer Not applicable AE, adverse event; SAE, severe adverse event; SOP, standard operation procedure. 1 Are there any inclusion and exclusion criteria in the study protocol. Does the inclusion criteria of prospective study reflect the voluntary participation of subjects in the study. Is the exclusion criteria a simple repetition of the inclusion criteria. 2 Is there any previous research data support for the calculation of sample size in confirmatory research.
Frontiers in Public Health frontiersin.org . /fpubh. . the standardized application of clinical diagnosis and treatment technology for frequently occurring diseases, chronic diseases, and difficult diseases and the promotion of research achievements. We included IITs that are (1) on human subjects, (2) with protocol attachment or published protocol, and (3) with enrolled patients study. Animal or vitro experiments, projects without data collection, and studies without protocols were excluded. Finally, a total of 257 IITs were included in our research, owing to four projects were excluded due to not human subjects, and another 11 were excluded due to actively apply for project termination.

Data extraction and quality assessment
Our previous study refers to the risk-based monitoring method advocated by the international community and globally. It explored a set of standard processes in the framework of clinical research and regulation, which mainly included preparation before quality assessment, self-assessment of the project team, centralized inspection, on-site inspection, report writing, and comprehensive assessment. Referring to the clinical research quality assessment process, our team provided midterm verification and quality control technical support for the quality assessment of the implementation process of IIT projects (21). 257 clinical research projects' quality assessment was also evaluated through the standard processes. Data were collected based on the review of research protocols, ethical approval, case report forms (CRFs), and informed consent submitted by researchers.
A nine-member committee is mainly responsible for data collection and access the indicators of IIT, including PM, DM, CRA, statistician, clinical expert, financial expert, scientific research management expert. PM was responsible for work coordination, DM was responsible for database building and management, two CRA were responsible for the collection of data on quality and ethical regulation, a financial expert was responsible for the collection of data on implementation of the project funds, a senior research manager was responsible for the collection of data on overall progress and enrollment progress of IITs, and the statisticians and clinical experts are responsible for the collection of data on scientific aspects which mainly assess level of evidence and appropriate research method of IITs. All text reviewers were trained in standard procedures to review research files.
For quantitative analysis, the following quality indexes were assessed: overall progress, enrollment progress, budget implementation rate. Each quality index was scored from 0 to 10. Overall progress was divided into not-enrolled, enrolled, treatment completed, follow-up completed, research type including RCT, Prospective cohort, Retrospective study, Real world research, Others. These statuses were given different scores. For quantifying data, each quality index was scored based on the components in research files proportionally. The total score for each index is 10 point. If any "No" and "Can't answer" were answered in each subindex, one point would be deducted. The minimum score of each index is 0.

Statistical methods
The REDCap [Research electronic data capture (http:// projectredcap.org)] was used to input data to control the quality of data. All statistical analyses were performed by R statistical software. Descriptive of basic characteristics of included IITs was presented by mean standard deviation or percentage, as appropriate. The structure validity of the index system of the quality evaluation of clinical research was tested by confirmatory factor analysis. Model fit was evaluated using the Tucker-Lewis Index (TLI), CMIN/DF the ratio chi-square (χ )/degrees of freedom (DF), Goodness-of-Fit Index (GFI), Comparative Fit Index (CFI), Standardized Root Mean Squares Residual (SRMR), and the Root Mean Square Error of Approximation (RMSEA). This index ranged from 0 to 1, the better the fit is, and it is generally believed that CFI and TLI should be >0.9, and GFI is at least >0.80 (22). RMSEA is the index of evaluation model fitting. SRMR index measures the fitting degree of the model by measuring the standardized difference between the observed correlations and the model implied correlations about variables. The closer it is to 0, RMSEA values as high as 0.07 were regarded as acceptable and SRMR should be <0.08 (23). It is considered that the model fits well (22). All statistical analyses were performed by using the R statistical software, "lavaan" package, and "semPlot" package (R version 3.5.3). Twosided P values of <0.05 were considered to indicate statistical significance. In Excel, the radar plot was generated by using the insert function.

Results
The framework of quality indexes for IITs We conducted two rounds of Delphi panels. The positive coefficient of the two rounds of experts in this study was 100% and the degree of expert authority was 0.932, indicating that the experts participated actively, showing a high degree of authority and a good effect of consultation. We revised some indices by summarizing and analyzing experts' opinions in the first round. For example, study protocol dissemination, one of the scientificity attributes, has been included. Experts believed that all prospective clinical trials should have an appropriate registry before the first participant is enrolled, and study protocol should be made public. At the end of the first round, we deleted two indicators and increased one indicator, and we modified the expression. In the second round of Delphi, we fed back the results of the first round consultation to the experts. Experts reached consensus on the revised indicators during the second round. Furthermore, the CFA method was used to analyze the reliability and validity of the indicators. The research finally identified the conceptual quality indicators for IITs which contain four themes: progress, quality, regulation, and scientificity. As shown in Table 1, 4 quality properties and 13 quality indices were developed through conducting clinical trials.

Validation of quality assessment tool of IITs
The preliminary quality indicators of quality assessment tools were further evaluated construct validity by using CFA (Figure 3). The critical quality properties and quality indexes between expertise and CFA are consistent. In this study, CMIN/DF, GFI, CFI, TLI, RMSEA, and SRMR indexes were selected as the indexes of the statistical model. It can be noticed in Table 3 that CMIN/DF was 1.868 <3, GFI, CFI, and TLI were all larger than 0.9, RMSEA was <0.07, and SRMR was <0.08, and it can be considered that the above quality assessment index extraction results were feasible. As is shown in Table 4, factor loadings were most of all P < 0.05 and ranged from a minimum of 0.306 to a maximum of 0.786. Meanwhile, the factor loadings between overall IITs quality and one-class index ranged from 0.425 to 0.967 with P < 0.05. CFA was used for the index screening of 257 IITs on 17 indices. The results of CFA suggested that the standardized factor loading of 17 indices were all statistically significant, and the factor loading all remained above 0.3 (23), and thus the preliminary entry screening did not delete any index.

Results of quality assessment of IITs
Radar graphing was used to display data. Radar plots had a series of spokes or rays arising from a central point, with each ray showing a different index, such as data management. Figure 4 illustrated that polygons were created with each spoke showing one of the secondary indexes and each point on the spoke reflecting the magnitude of the mean results, and different colors region represent the different one-class index. Among 257 IITs, data management was well-presented with an average score of 9.40, followed by quality control of 9.36. Enrollment progress and budget implementation rate scored low, which were 7.30 and 5.91, respectively. RCT, prospective cohort and retrospective performed similarly in subject management, appropriate research method, and quality control. However, RCT had lower scores than prospective cohort and retrospective study in enrollment progress (7.02 vs. 7.43, 9.63, respectively).

Discussion
Our study proposed a proactive quality assessment consideration for IITs which consisted of four aspects: progress, quality, regulation, and scientificity. A total of 257 IITs were included for the validation of quality indicators. We further confirmed the structural validity of the critical quality properties and quality indexes as latent variables. According to the results of CFA, CMIN/DF, GFI, CFI, TLI, RMSEA, SRMR, and other indicators used in the model test all met the requirements. The CFA model revealed strong positive links from four quality indexes to the overall quality of ITTs.
The panorama quality assessment tool can effectively evaluate the quality of research, and help find major bias. For the management side like National Institutes of Health, the assessment tool can effectively improve the efficiency and effectiveness of funding. With the help of this tool, funding agencies can better decide which project should be supported or terminated. Assessment tool is very important to manage and improve the quality of clinical trials. Quality assessment standard processes in the framework of clinical research and regulation follow international guidelines. Traditionally, the International Conference on Harmonization-Good Clinical Practices (ICH-GCP) described two verification activities: quality control and quality assurance, with the aims to protect the rights and wellbeing of subjects and ensure protocol compliance and data integrity. FDA (17) and EMA (18) guidelines both issued in 2013, ICH GCP guidelines (16) issued in 2016 and NMPA GCP (24) issued in 2020 suggested focusing on critical data and critical processes, and encouraged to adopt risk-based approaches to monitor clinical trials. A series of research studies reported that risk-based monitoring had the potential to make trials more efficient and reduced costs (25,26). In this study, risk-based approach was adopted to identify critical data of IITs and improve the capacity of self-regulate overall quality. Various research types of IIT projects will increase the difficulty of quality evaluation. Standardization of practices in monitoring activities will be a suitable method for the management of IITs.
We further confirmed the structural validity of the IITs quality assessment tools by CFA. The model also presented the importance of progress and scientificity. Progress, measured jointly by overall progress, enrollment progress, budget implementation rate, exerted direct and indirect effects on the overall quality of IITs in our theory which was confirmed Frontiers in Public Health frontiersin.org . /fpubh. .  by the model. Poor recruitment of participants is the most common reason for the RCT discontinuation, which reflects a large waste of scarce research resources (27). In this study, enrollment progress and budget implementation rate scored 7.30 and 5.91 respectively, which were rather low. The reason for the low recruitment progress of subjects may be linked with funding, design, recruiter, or participant (28). In addition, there were differences in the progress of subject recruitment among different research types, and the lowest recruitment progress is RCT. Scientificity, measured jointly by level of evidence, appropriate research method, and study protocol dissemination had direct and indirect impacts on the overall quality of IITs in our theory which was confirmed by the model. Our study found that almost half (117/257) of the research protocol adjusted during the implementation process may be associated with poor design. It is important that all research findings, including negative and inconclusive results are reported transparently and made publicly available in order to avoid unnecessary duplication of research or biases in the clinical knowledge base (8). While our study found . /fpubh. .   (29), with almost half of the studies remaining unpublished years after completion may be aroused by "a lack-of-time or low priority, " followed by "results not important enough" and "journal rejection" (30). Different from previous assessment tools like RACT, which is more suitable for ISTs. The panorama tool developed in this study was optimized for funding agencies to assess the quality of ongoing IITs. For ISTs, the responsibility to avoid failure due to unsatisfactory progress or scientificity is mostly up to the industrial sponsor. Therefore, the regulatory department can focus on the ethics and quality aspects of the studies. However, for IITs, funding agencies should take progress and scientificity into consideration that the resources will be used more efficiently. Therefore, unlike the traditional point that the most important aspects of clinical trials are subject safety and rights plus data quality, this tool also emphasized the importance of progress and scientificity. Also, the tool can be used to assess the quality of both study design and implementation, and can be used throughout the whole clinical research, regardless of the research types. What's more, a little different from other risk based monitoring tools, this tool not only can find risks and determine the monitoring methods, also can be used to compare the quality of several clinical studies.
Our study had several advantages and limitations as well. First, the samples used to confirm the CFA were collected from ongoing studies, and no previous research was identified to discuss the ongoing studies. Our study developed the tool can objectively reflect the current research status, regardless of the research types. Second, the sample size was up to standard. We collected more than 200 samples for CFA and while CFI is a non-centrality parameter-based index designed to overcome the limitation of sample size effects (31). Third, our study adopted a risk-based monitoring method to identify critical data and processes, which was in line with international Frontiers in Public Health frontiersin.org . /fpubh. . trends and saved resources. The quality assessment tool for IITs enabled us to evaluate the overall quality of IITs and helped refine quality practices in IITs. However, we only included the clinical research projects in Shanghai hospitals. Further research is needed to confirm this tool in more general scenarios.

Conclusion
The results of critical quality properties and quality indexes between expertise and confirmatory factor analysis were basically consistent, indicating applying this panoramic quality assessment tool for overall quality evaluation of IITs is feasible and validated. This panorama tool can enable project management departments to effectively and dynamically manage the quality of their studies, and can timely and dynamically find errors, take actions to prevent major bias. Furthermore, the project management departments will be able to terminate the "low-quality" project in advance, and provide rolling support for the "high quality" project based on the situation of the projects. It is hoped that this tool can provide project management departments with resources for effective and dynamic management of researches and avoid waste of resources, as well as a manner to improve the quality of IITs in the future.

Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author. . /fpubh. .