Impact Statements

  • Specific dimensions of quality related to clinical pharmacokinetic research include study design, conduct, analysis, results validity, clinical relevance, and quality of reporting.

  • Developing a critical appraisal tool to evaluate clinical pharmacokinetic studies will help end-users to apply evidence-based medicine to clinical practice and future research.

  • The clinical pharmacokinetic critical appraisal tool (CACPK) developed in this study consisted of 21 items aimed at helping end-users to appropriately assess the quality of pharmacokinetic studies.

  • The currently developed CACPK tool shall undergo periodic improvement in order to keep pace with changes in the literature and to maintain the practical benefits to the field of clinical pharmacokinetics.

Introduction

The practice of evidence-based medicine (EBM) integrates individual clinician’s expertise and experience with the best available clinical evidence from empirical research [1, 2]. Central to the practice of EBM is the critical appraisal of published literature, in which the relevance, quality, and the trustworthiness of a published study is systematically assessed [2]. The critical appraisal process helps in assessing the validity, reliability, and quality of the published scientific knowledge. Therefore, several generic and specific critical appraisal tools for varying study designs were developed. Design-specific critical appraisal tools contain items that critique the methodological quality of the study design [3]. Generic critical appraisal tools help in appraising quantitative and qualitative studies in general [4]. Accordingly, the critical appraisal process aids in assessing the quality of the study results, and how they are interpreted and applied in health policymaking, therapeutic decision, and clinical practice [5]. Therefore, selecting the most appropriate critical appraisal tool is essential for the application of evidence-based practice [6].

The application of pharmacokinetic principles in clinical settings is considered an integral part of pharmaceutical care provision by the pharmacist [7]. Clinicians aim to improve a patient’s response to the drug dosage regimen and to minimize toxicity by designing an individualized dosage regimen through the application of clinical pharmacokinetic concepts [8]. This process is aided by literature review of available evidence extracted from clinical pharmacokinetic studies, which have to meet high standards of quality to ensure the delivery of safe and effective therapeutic regimen [9].

There are many developed and published critical appraisal tools and reporting quality checklists to guide researchers to assess the quality of the published research. Reporting checklists, however, are not synonymous to critical appraisal tools. A study may meet the required aspects for reporting, but may fail to meet the expected quality standards. Furthermore, some reporting checklists may not assess important dimensions of quality related to the study design, conduct, analysis, clinical relevance, and result validity [9]. Additionally, currently published critical appraisal tools are not highly specific to determine the methodological quality and validity of clinical pharmacokinetic studies. Most of the available tools lack specific items that help in analyzing the published articles in depth [10].

To our knowledge, two studies were conducted to assess the quality of reporting clinical pharmacokinetic studies [11, 12]. The first study was a systematic review, which evaluated the quality of reporting of pharmacokinetic studies of antibiotics in patients with sepsis receiving continuous renal replacement therapy [11]. In this systematic review, the researchers found that none of the identified articles during their systematic search reported the full set of parameters that help end-users interpret the reported results. Furthermore, 20% of the published pharmacokinetic trials did not contain the fundamental pharmacokinetic parameters [11]. Consequently, reporting guidelines for clinical pharmacokinetic studies (The ClinPK Statement) were issued to guide investigators in reporting clinical pharmacokinetic studies. The developers designed a yes/no checklist that was composed of 24 items to guide researchers in reporting the minimum required information in clinical pharmacokinetic studies [12]. While offering a valuable guideline for reporting the findings of clinical pharmacokinetic studies, The ClinPK Statement has its drawbacks and did not cover some important dimensions of quality. Another tool, Grading and Assessment of Pharmacokinetic-Pharmacodynamic Studies (GAPPS), was recently published to assess the strength of evidence extracted from pediatric PKPD antibiotic studies [13].

A critical appraisal tool with broad application that aids clinicians in appraising and assessing the quality of published clinical pharmacokinetic studies does not exist. Developing an understanding of quality markers for clinical pharmacokinetic studies that clinicians consider important would aid in the creation of a tool aimed at assessing the quality of published clinical pharmacokinetic studies.

Aim

We aimed to achieve an expert consensus regarding the quality markers of clinical pharmacokinetic studies with an attempt to develop a critical appraisal tool through a modified Delphi technique.

Ethics approval

This study was approved by the Qatar University Institutional Review Board: QU-IRB 970-E/18. Data were analyzed using SPSS version 24 (IBM SPSS® Statistics for Windows, version 24; IBM Corp, Armonk, NY, USA).

Method

Use of pre-selected set of preliminary quality markers

A pre-selected set of preliminary quality markers of clinical pharmacokinetic studies identified systematically [14] was used to build a questionnaire, comprising potential identified candidate items into a design-specific critical appraisal tool. These items were categorized into relevant study sections, including title, abstract, background, methods, results, discussion, conclusion and other.

Sampling and selection of Delphi experts

A purposive sampling method was used to select participants who met the study’s eligibility criteria. Potential participants were approached as expert panelists if they were considered as any of the following:

  • academic experts with a designation that reflects their direct involvement in clinical pharmacokinetics as evidenced in their research and teaching portfolios.

  • clinicians who had experience in the application of clinical pharmacokinetic principles or the provision of therapeutic drug monitoring in their clinical practice. These individuals should have experience in interpreting the findings of clinical pharmacokinetic studies and applying these to their patients.

  • pharmaceutical industry researchers with years of experience in relevant positions involving clinical pharmacokinetic research.

  • individuals in regulatory bodies who assess clinical pharmacokinetic studies when making decisions for their respective health authorities, e.g. FDA.

Potential participants were requested to state their area of expertise related to clinical pharmacokinetics, number of years in the field of clinical pharmacokinetics, current geographical location of practice and a global self-assessed competence in the field of clinical pharmacokinetics (none, little, average, advanced).

Modified Delphi survey

A modified Delphi process was utilized via an online survey platform (SurveyMonkey®) in order to build consensus amongst experts regarding each item considered for inclusion in the tool. An inventory of quality markers to evaluate pharmacokinetic studies [14] was used to identify potential items through a systematic approach. These items were revised scientifically and linguistically, reworded, and reduced after several iterations with the research team members prior to being sent to the study participants. A large number of participants could be included in the rounds through using modified Delphi design because of the lack of geographical restrictions. Participants could express their opinions freely, as their answers were anonymous. Therefore, a dominance that might occur during face-to-face interactions and any bias introduced by moderators were avoided. Panelists in this study were recruited from different countries and practice settings like academia, industry, clinical practice, and regulatory authorities. Thus, each participant could share their perspective in the field of clinical pharmacokinetics and help in generating new ideas that helped in broadening the knowledge base of other participants.

Modified Delphi process

Round 1 of this modified Delphi process utilized a questionnaire comprising potential identified candidate items. Experts were presented with individual candidate items and were requested to state their level of agreement that the item should be included using a 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree). For each candidate item, participants were given the option to provide free-text comments to support their rating or to suggest changes to the item. Upon completion of the potential items generation, participants were given the option to nominate additional items for consideration.

Consensus was set a priori at 75% agreement with any one of the available outcomes (inclusion, exclusion, modification). Inclusion was met if 75% or more of the participants rated an item as agree or strongly agree. Exclusion was met if 75% or more of participants rated an item as disagree or strongly disagree. Item modification was considered if neither inclusion nor exclusion were met after which the item was modified based on participants’ feedback. If after a modification, an item still did not meet the requirements for inclusion or exclusion and the percentage change in consensus was less than 15%, it was excluded from the final tool to be used for critical appraisal of clinical pharmacokinetic studies (CACPK). For each additional Delphi round following round 1, participants were given the blinded aggregate comments from all other participants for each item. As the aim of the study was to create a tool to be used to assess the quality of clinical pharmacokinetic studies, the researchers sought to gauge the participants’ opinion of an effective means of rating each item included in the tool. Consequently, an additional section of the survey asked participants to rate various scales of measurement that could be used to assess each item in the finalized tool. These initially included options of “yes/no” and a 4-point scale ranging from “poor, fair, good, to excellent”. Authors were also open to additional options presented during the modified Delphi. The experts were also provided a free-text box to suggest additional rating systems. Participants were given four weeks to complete each Delphi round with reminder emails being sent after three weeks. All participants’ responses were kept anonymous from the other participants.

Pilot testing

The CACPK final tool was applied to 30 clinical pharmacokinetic studies in order to test validity and feasibility of the tool in evaluating published literature. Ten studies were purposively selected each from three journals that focus on clinical pharmacokinetics (Clinical Pharmacokinetics, International Journal of Pharmacokinetics, Journal of Pharmacokinetics and Pharmacodynamics). Articles included randomized clinical pharmacokinetic trials, evaluations of drug interactions, population pharmacokinetic studies, and bioequivalence studies. Studies including less than five participants were excluded from the pilot testing. Application of the quality tool was performed by four end-users (AS, SP, OR, KW) in duplicate. Included articles were divided equally at random amongst raters so that each rater evaluated 15 articles and each article was evaluated twice. Raters were also asked to record the time to complete the quality assessment of each article evaluated. Outcomes of ‘No, I don’t know, and Not applicable’ were grouped together. Cohen’s Kappa was used to assess level of agreement between raters [15]. Kappa values of less than 0 were rated as less than the chance of agreement; 0.01–0.20, slight agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, substantial agreement; 0.81–0.99, almost perfect agreement. Average time to complete each assessment was assessed for feasibility of the tool.

Results

Participants

One hundred and nineteen potential expert panelists, identified by the research team or through snowballing, were invited to take part in this study. Personal invitations were sent via email. Twenty-five panelists agreed to participate in the modified Delphi process. The sociodemographic characteristics of the panelists of the modified Delphi are shown in Table 1.

Table 1 Sociodemographic characteristics of modified Delphi panelists
Full size table

The average level of experience of the participants in the field of clinical pharmacokinetics was 8.5 years with 56% self-identified as clinicians with relevant practice experience in clinical pharmacokinetics. Around 96% of the participants rated themselves with an average or advanced level of competence in clinical pharmacokinetics.

Consensus through modified Delphi rounds

Three rounds of the modified Delphi were conducted. Response rates for each round were 24/25 (96%), 23/25 (92%), and 15/25 (60%), respectively.

Forty items were identified through systematic search of the literature and were included in the first round of the Delphi. Two additional control items that were not deemed relevant to clinical pharmacokinetics were included to gauge the integrity of the participant responses for a total of 42 items (Supplement 1). At the end of round 1, the panel reached consensus on inclusion of 12 items (28.6%). These items were included in the CACPK final tool and were not entered into round 2. Two items met consensus for exclusion (4.8%). Both control items met consensus for exclusion and were not included in any additional rounds of the Delphi. Furthermore, two additional items were added based on participant suggestions/comments. The remaining 26 items were amended based on participants’ comments and were re-entered into round 2 (Fig. 1).

Fig. 1
figure 1

The modified Delphi flow chart of all rounds

Full size image

In round 2, 28 items were presented to the participants for consideration. At the end of the round, 6 items (21.4%) reached consensus for inclusion in the CACPK final tool, while 19 items (67.9%) reached consensus for exclusion. The remaining three items (10.7%) were reformulated based on participant’s feedback and were included in round 3.

Round 3 comprised of 3 items in which all (100%) reached consensus for inclusion into the CACPK final tool. Consensus rates for each of the included item along with the round of the modified Delphi in which they were approved are summarized in Table 2. Participant consensus regarding the choice of rating of each item in the CACPK final tool was for items to be rated as ‘Yes, No, I don’t know, and Not applicable’.

Table 2 Consensus rate of the items included in the final tool for Critical Appraisal of Clinical Pharmacokinetic (CACPK) studies
Full size table

The CACPK final tool included 21 items that met consensus for inclusion (Table 3). The tool was divided into five sections: background, design, sampling, applied statistics, and results. The final rating scale that was used to help the end-users to assess the quality of published clinical pharmacokinetic studies was “Yes, No, I Do Not Know, and Not Applicable”.

Table 3 Final tool for Critical Appraisal of Clinical Pharmacokinetic studies (CACPK)
Full size table

Pilot testing

Thirty recently published articles focused on clinical pharmacokinetics were assessed for quality (Supplement 2) using the quality assessment tool developed through the modified Delphi (CACPK final tool). A majority of items (12/21) were scored at fair to moderate agreement. Of note, items 13 (were sample storage conditions appropriate and described in a manner that could be accurately replicated?), 16 (were the essential pharmacokinetic parameters required to make the results applicable in clinical settings addressed?), 18 (were the chosen statistical tests and software to perform the statistical analysis appropriate to achieve the study objectives?) and 21 (were appropriate summary statistics to describe centrality and variance used to document the pharmacokinetic results?) scored substantial to almost perfect agreement. Kappa values could not be calculated for items 2 and 5 as all respondents answered ‘yes’ with 100% agreement. Items 3 and 10 scored less than the chance of agreement.

Applying this CACPK final tool to clinical pharmacokinetic studies took an average of 28.5 min (range 14–90 min). No raters expressed any difficulty in understanding any of the items or applying these to the studies.

Discussion

Statement of key findings

The CACPK tool (Supplement 3) developed in this study offers new understanding of quality markers that should be utilized for appraising published clinical pharmacokinetic studies via pharmacokinetics experts consensus. Together with the CACPK tool, two additional appendices (Supplements 4 and 5) were created in case further clarification is needed about items 7 and 15 in the tool, respectively. This tool with its two appendices guide researchers through answering 21 questions to determine the quality of published clinical pharmacokinetic studies. The tool was developed in a format similar to other critical appraisal tools like a measurement tool to assess systematic reviews (AMSTAR) to facilitate the appraising process and was composed of four sections: appraising background, appraising study design and experimental methods, appraising applied statistics, and appraising results [16].

Strengths and weaknesses

The development of a critical appraisal tool for clinical pharmacokinetic studies adds much value to the existing literature as it complements currently available checklists by assessing dimensions of quality of clinical pharmacokinetic studies. This tool, with further testing of reliability and validity, can be potentially used to guide clinicians and policymakers to evaluate the quality of published articles to take clinical decisions and develop policies by applying EBM. Furthermore, stakeholders who work in the academic sector can use it to teach students how to appraise this type of study.

Despite these strengths, we need to acknowledge the limitations of this study. Participants’ identification was anonymous as researchers, which prevented the ability to correlate responses to professional experience. Furthermore, we received a declining response rate with each round of the survey leading to overemphasized response from a smaller number of participants in the latter rounds of the study.

Interpretation

The aim of developing the CACPK tool was to assess the quality of studies, which has several dimensions including study design, conduct, analysis, clinical relevance, results validity and quality of reporting. While other clinical pharmacokinetic guidelines have focused on assessing the quality of reporting of clinical pharmacokinetic research, these tools do not directly address whether the methods used were of high quality. This tool thus assesses the appropriateness of the execution of methods used in carrying out the desired clinical pharmacokinetic analysis.

Pilot testing of the clinical pharmacokinetic critical appraisal tool amongst four individuals resulted in a majority (12/21) of items on the tool scoring fair to moderate agreement. While these four individuals did have considerable experience in the application of clinical pharmacokinetics, this result may not be replicated if clinicians with little to no understanding of clinical pharmacokinetic principles use this tool. Application of this tool in assessing the quality of a clinical pharmacokinetic study did however appear to be reasonably feasible in regards to time requirement with the average study taking under 30 minutes to appraise.

To our knowledge, this study is among the few ones to identify quality markers of clinical pharmacokinetic studies for developing a specific critical appraisal tool with broad application. One of the main strengths of this study was the robustness of the methodology used to develop clinical pharmacokinetics critical appraisal tool. In the modified Delphi process, we were keen to recruit experts who represent all clinical pharmacokinetic stakeholders (clinicians, researchers, individuals in academic and industrial sectors and policymakers). The inclusion of different stakeholders allowed us to enrich our tool with different perspectives from different end-users.

Further research

Further research can be conducted to further assess the validity, reliability, and overall utility of the tool. Research should also be conducted to determine how the presence or absence of most important markers of quality for clinical pharmacokinetic studies affects study results and interpretation. We acknowledge that the field of clinical pharmacokinetics is vast and that end-users of clinical pharmacokinetic research extends to a vast array of clinicians. It is therefore unlikely that we included expert panelists from all areas relevant to clinical pharmacokinetics.

Conclusion

This study aimed to achieve expert consensus regarding pre-identified set of quality markers for the appraisal of clinical pharmacokinetic studies. Through this modified Delphi process, a list of questions gauging the overall quality of clinical pharmacokinetic studies was developed that allowed for critical appraisal. Work presented in this study provides a critical appraisal tool (CACPK) with broad application to clinical pharmacokinetic studies.