Skip to main content
SpringerLink
Log in

Section Original article

Advantages and limitations of Twin Assessment of Clinical Trials (TACT)

  • Original Article
  • Published:
Journal of Public Health Aims and scope Submit manuscript

Abstract

Background

Considerable time and energy are expended in the scientific community to discuss the validity, importance, and applicability of the results of clinical trials. Depending on the goals, perspectives, and other motivating factors, protagonists and skeptics come to different conclusions, even when using the same methods and tools for critical appraisal. The aim of this study was to complement existing methods and tools with minor modifications to provide a prototype instrument that generates commonly accepted versions of critical appraisals.

Methods

As a pilot experiment, one university-based and one industry-based referee independently completed the twin assessment of five trials published in well-recognized journals. They identified the study questions, defined the simplest, i.e., ideal, study designs to answer these questions, and checked eight validity criteria. Identical positive or negative answers of both referees increased or decreased the validity score. A maximum of two disagreements (0 score) was allowed. This procedure, which had been tested by two referees in a pilot experiment, was repeated with 19 third-year medical students and their supervisor at the Universidade Federal Fluminense, Niterói/RJ, Brasil. Four students each played the roles of the industry-based and university-based referees and finally recorded their consensus.

Results

The two referees of the pilot experiment agreed in all but one answer to the five investigated publications. The points of criticism differed in various papers. The consensus reached by the students considerably differed from the consensus reached by the referees.

Conclusions

A consensus score generated by two referees or by two groups of students is feasible, but the achieved result is not necessarily reproducible. The critical appraisal of the study question in connection with the applied study design deserves special attention. It is time consuming but possible to identify and describe the possible flaws in the design conduct and report of clinical trials, but it is unlikely to reach a reproducible interpretation. These data indicate the problems with even evidence-based assessments and appraisals: the assessments may well be reproducible, but not the appraisals. Quality scores that include also the appraisal may therefore be interpreted with caution. Appraisals or quality scores may be used for interim decisions until data are available that confirm under real-world conditions what was predicted by the results generated under ideal but artificial conditions of a clinical trial.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Barry MJ (2009) Screening for prostate cancer—The controversy that refuses to die. New Engl J Med 360:1351–1354

    Article  CAS  PubMed  Google Scholar 

  • Coppin C, Porzsolt F (2003) Kidney Cancer. Evidence-Based Oncology. In: William C (ed) Evidence-Based Oncology. BMJ Books 333–345

  • Darmoni SJ, Haug MC, Lukacs B, Biossel JP (2001) Quality of health information about depression on internet. Level of evidence should be the gold standard. BMJ 322:1367

    CAS  Google Scholar 

  • Dautzenberg B, Arriagada R, Chammard AB, Jarema A, Mezzetti M, Mattson K, Lagrange JL, Le Pechoux C, Lebeau B, Chastang C (1999) A controlled study of postoperative radiotherapy for patients with completely resected nonsmall cell lung carcinoma. Groupe d'Etude et de Traitement des Cancers Bronchiques. Cancer 86:265–273

    CAS  Google Scholar 

  • Delgado Rodriguez M, Ruiz-Canela M, de Irala-Estevez J, Llorca Diaz J (2001) Martinez-Gonzalez MA (2001) Differences in the quality of Spanish clinical trials published in international periodicals and of the ones presented in general medicine periodicals with wide readership. Rev Clin Esp 201:437–443

    CAS  PubMed  Google Scholar 

  • Forestier R, Francon A, Graber-Duvernay B (2005) Validity parameters of clinical trial and their influence on evidence based medicine conception: a review. Ann Readapt Med Phys 48:250–258

    CAS  PubMed  Google Scholar 

  • GRADE Working Group (2004) Grading quality of evidence and strength of recommendations. BMJ 328:1490–1494. doi:10.1136/bmj.328.7454.1490

    Article  Google Scholar 

  • Heart Protection Study Collaborative Group (2002) MRC/BHF Heart Protection Study of cholesterol lowering with simvastatin in 20,536 high-risk individuals: a randomised placebo-controlled trial. Lancet. 360:7–22

    Article  Google Scholar 

  • Jørgensen A, Hilden J, Gøtzsche PC (2006) Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ 333:782–785

    Article  PubMed  Google Scholar 

  • Knipschild P, Leffers P, Feinstein AR (1991) The qualification period. J Clin Epidemiol 44:461–464

    Article  CAS  PubMed  Google Scholar 

  • Mehta SR, Yusuf S, Peters RJG, Bertrand ME, Lewis BS, Natarajan MK, Malmberg K, Rupprecht H-J, Zhao F, Chrolavicius S, Copland I, Fox KAA (2001) Effects of pretreatment with clopidogrel and aspirin followed by long-term therapy in patients undergoing percutaneous coronary intervention: the PCI-CURE study. The Lancet 358:527–533

    Article  CAS  Google Scholar 

  • Meunier PJ, Roux C, Seeman E, Ortolani S, Badurski JE, Spector TD, Cannata J, Balogh A, Lemmel E-M, Pors-Nielsen S, Rizzoli R, Genant HK, Reginster J-Y (2004) The Effects of Strontium Ranelate on the Risk of Vertebral Fracture in Women with Postmenopausal Osteoporosis N Engl J 350:459–468

    CAS  Google Scholar 

  • Miettinen OS, Henschke CI (2001) CT screening for lung cancer: coping with nihilistic recommendations. Radiology 221:592–596

    Article  CAS  PubMed  Google Scholar 

  • Mills E, Loke YK, Wu P, Montori VM, Perri D, Moher D, Guyatt G (2004) Determining the reporting quality of RCTs in clinical pharmacology. Br J Clin Pharmacol. 58:61–65 and 58:102

    Google Scholar 

  • Nothardt (2007) Validity of Clinical Trials in Homeopathy included in Systematic Reviews. Thesis Medical Faculty University of Ulm, 2007

  • Obremskey WT, Pappas N, Attallah-Wasif E, Tornetta P 3 rd, Bhandari M (2005) Level of evidence in orthopaedic journals. J Bone Joint Surg Am 87:2632–2638

    Article  PubMed  Google Scholar 

  • Ohmann C, Albrecht J. Lessons to be learned for gastroenterology from recent issues in clinical trial methodology (2000) Can J Gastroenterol 14: 293–298

  • Phillips B, Ball C, Sackett D, Badenoch D, Straus S, Haynes B, Dawes M. Levels of evidence. Homepage des Oxford-Centre for Evidence-Based Medicine. http://www.cebm.net/index.aspx?0=1025

  • Piccart-Gebhart MJ, Procter M, Leyland-Jones B, Goldhirsch A, Untch M, Smith I, Gianni L, Baselga J, Bell R, Jackisch C, Cameron D, Dowsett M, Barrios CH, Steger G, Huang CS, Andersson M, Inbar M, Lichinitser M, Lang I, Nitz U, Iwata H, Thomssen C, Lohrisch C, Suter TM, Ruschoff J, Suto T, Greatorex V, Ward C, Straehle C, McFadden E, Dolci MS, Gelber RD; Herceptin Adjuvant (HERA) Trial Study Team (2005) Trastuzumab after adjuvant chemotherapy in HER2-positive breast cancer. N Engl J Med 353:1659–1672

    Article  Google Scholar 

  • Porzsolt F, Kumpf J, Coppin C, Pöppel E (2003a) Stringent application of epidemiologic criteria changes the interpretation of the effects of immunotherapy in advanced renal cell cancer. In: William C (ed): Evidence-Based Oncology. BMJ Books 34–38

  • Porzsolt F (2003b) Klinische Ökonomik. Die ökonomische Bewertung von Gesundheitsleistungen aus der Sicht des patienten. In: Porzsolt F, Williams AR, Kaplan RM (Hrsg.) Klinische Ökonomik. Effektivität and Effizienz von Gesundheitsleistungen. ecomed Verlagsgesellschaft 17–40

  • Porzsolt F, Kajnar H, Awa A, Fässler M, Herzberger B (2005) Validity of original studies in Health-Technology Assessment (HTA) reports: Significance of standardized assessment and reporting. Int J Technol Assess Health Care 21:1–4

    Article  Google Scholar 

  • Straus SE, Richardson WS, Glasziou P, Haynes RB (2005) Evidence-Based Medicine. How to Practice and Teach EBM. 3rd edition. Elsevier Churchill Livingstone, Edinburgh, London New York

  • van Gijn J (1999) From therapeutic trials to current practice. Rev Neurol (Paris) 155:708–712

    Google Scholar 

Download references

Acknowledgement

We are grateful to Eva Greimel (Department of Gynecology, Medical University of Graz), Martin Eisemann (Department of Medical Psychology, University of Tromsø) and Jörg Sigle (Clinical Economics, University of Ulm) for their contributions to the meeting at Kohldorf/Steiermark, Austria, where the pros and cons of disclosing the details of the publications were discussed.

Conflict of interest

Franz Porzsolt is a consultant for Sevier Deutschland.

Author information

Authors and Affiliations

  1. Clinical Economics, University of Ulm, Frauensteige 6, 89075, Ulm, Germany

    Franz Porzsolt

  2. SHUFFLE, Dept Physiology, Federal University Fluminense, Niterói, Rio de Janeiro, Brasil

    Ian Curi Bonotto de Oliveira Costa & Tania Gouvêa Thomaz

Authors
  1. Franz Porzsolt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ian Curi Bonotto de Oliveira Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tania Gouvêa Thomaz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Franz Porzsolt.

Additional information

Members of SHUFFLE (Students Help Universidade Fedeal Fluminense in Literature Export) are Prof. Dr.Tania G. Thomaz, MD, Ian C.B.O. Costa (manager), Priscila F. de Amorim, Rafael C. Astorga, Renato C. Branco, Igor Brandão, Gustavo M. Cardoso, Lara Dan, Carolina D. Gonçalves, Thiago Gonçalves, Tabata Graciolli, Davi L. Grilo, Ingrid E. Hinden, Pedro M. Lisboa, Gabriela M. Machado, Pascale G. Massena, Lauanny A. Pereira, Rafael Perez, Natalia C.Z. Silva, and Fábio J.S. Souza.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Porzsolt, F., Costa, I.C.B.d.O. & Thomaz, T.G. Section Original article. J Public Health 17, 425–432 (2009). https://doi.org/10.1007/s10389-009-0283-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10389-009-0283-4

Keywords

Search

Navigation