Deficiencies of effectiveness of intervention studies in veterinary medicine: a cross-sectional survey of ten leading veterinary and medical journals

The validity of studies that assess the effectiveness of an intervention (EoI) depends on variables such as the type of study design, the quality of their methodology, and the participants enrolled. Five leading veterinary journals and 5 leading human medical journals were hand-searched for EoI studies for the year 2013. We assessed (1) the prevalence of randomized controlled trials (RCTs) among EoI studies, (2) the type of participants enrolled, and (3) the methodological quality of the selected studies. Of 1707 eligible articles, 590 were EoI articles and 435 RCTs. Random allocation to the intervention was performed in 52% (114/219; 95%CI:45.2–58.8%) of veterinary EoI articles, against 87% (321/371; 82.5–89.7%) of human EoI articles (adjusted OR:9.2; 3.4–24.8). Veterinary RCTs were smaller (median: 26 animals versus 465 humans) and less likely to enroll real patients, compared with human RCTs (OR:331; 45–2441). Only 2% of the veterinary RCTs, versus 77% of the human RCTs, reported power calculations, primary outcomes, random sequence generation, allocation concealment and estimation methods. Currently, internal and external validity of veterinary EoI studies is limited compared to human medical ones. To address these issues, veterinary interventional research needs to improve its methodology, increase the number of published RCTs and enroll real clinical patients.


Criteria for journal inclusion 25
To be eligible for inclusion in the study the journals must be in English, must have a 26 broad scope (i.e., general and internal medicine) and must have been relevant in the 27 field for a certain period. 28 The "VETERINARY SCIENCES" category of the 2013 ISI Journal Citation 29 Report was sorted by decreasing impact factor. All the journals that focused on sub-30 specialties (e.g., Veterinary Microbiology, Veterinary Parasitology, etc.) or in non-31 English language were excluded. To endorse historical journals, all the journals that in  (Table 1). The following data items were recorded during the data extraction procedures. 52

Number of full original articles 53
The total number of full original articles was recorded because in previous studies in 54 other specialties [3,4] the prevalence of RCTs was provided as: RCTs/All published 55

articles. 56
Reports eligible as full original articles: primary research, including subgroup 57 analysis or follow-up of previous articles; case series, defined as original reports 58 including more than one patient. 59 Reports not eligible as full original articles: Single case reports; Secondary 60 research, including systematic reviews and non-systematic reviews; Qualitative

Number of EoI articles that described surgical interventions 90
The type of intervention was categorised in surgical/non-surgical, as 91 researches of surgical interventions face different challenges regarding several 92 aspects, including study design [7]. EoI articles were considered "surgical" when (1) 93 the intervention required cutting of the skin. Needle-related procedures (e.g., 94 amniocentesis, etc.) were not considered surgical procedures; (2) the difference 95 between the control and the experimental group was in the presence, type or technique 96 of the surgical procedure. If the difference between the control and the experimental group was a medication given after/before a surgical procedure the trial was not 98 interventions described as randomised were included [9]. A study was classified as "a 105 RCT" when (1) at least two interventions were compared; (2) and randomisation was 106

mentioned. 107
Studies based on RCTs that are not the primary outcomes of RCTs, subgroup 108 analyses or long-term outcome of previously published RCTs were also considered 109 RCTs if randomisation was maintained. Articles were included when they reported on 110 "randomly" allocated interventions, even when their actual allocation was 111 "nonrandom" (e.g., alternation, date of admission, etc.). Crossover studies (including 112 Latin square design studies) were considered RCTs if the patients were randomly 113 assigned to the treatment groups. If the word "random", "randomly", "randomised" or 114 "randomised" was not used to describe allocation of the patients to the treatment, the 115 study was not considered a RCT. 116 117

Number of RCTs that included real patients 118
We evaluated if RCTs involved real clinical patients or non-patients, i.e., 119 voluntary individuals or experimental animals. Real clinical patients were defined as 120 "the population that presents the condition that needs to be treated or prevented and 121 that will benefit of the intervention once established". Articles were considered to include real clinical patients when these individuals or animals: (1) suffered from a 123 spontaneous disease; and (2) were exposed to real-life conditions. Only animals kept

Assessment of reporting of key methodological domains in RCTs 136
The following protocol was applied for the assessment of reporting key 137 methodological domains in RCTs. Two operators (ND, LNPC) independently 138 assessed the RCTs. In case of disagreement, an arbiter was consulted (RMR). Firstly, 139 the materials and methods section was thoroughly read and relevant information was 140 highlighted. Then, key words were searched using the search function of Portable 141 Document Formats (PDF)s to find additional information that was not reported in the 142 materials and methods section. The search words included: "power", "sample" and 143 "size" for power calculation; "primary", "main", "outcome" and "endpoint" for 144 primary outcome; "random", "allocat" for randomisation and allocation concealment; 145 "blind", "mask", "aware", "know", "inform" for blinding domains; "95", "CI" and 146 "interval" for effect size estimation; "intent", "analysis" and "attrition" for handling of attrition. To avoid inappropriate exclusion of pertinent items, all grammatical 148 derivatives of these search terms were applied during these searches. Finally, each 149 methodological domain was scored "yes" if adequately reported, or "no" if not 150 adequately reported. 151 Evaluation of additional data than the published article -To avoid 152 inappropriate exclusion of eligible articles, full-texts of protocols, supplements, and 153 previous or accompanying manuscripts, which were explicitly mentioned in the main 154 text were also assessed. In the case that no references to supplementary material were 155 present in the main text, only the published report was assessed. 156 157

Key methodological domains 158
We evaluated the following key methodological domains [10]: 159 Primary outcome-Authors explicitly reported a primary outcome in the 160 published article. If a primary outcome was not explicitly described, we considered 161 the outcomes reported in the sample size estimation. When a primary outcome was 162 not explicitly specified in the article or sample size calculation the paper was 163 classified as "not reporting a primary outcome". We recorded whether the primary 164 outcome was retrieved from the power calculation or from a proper sentence. 165 Power calculation-Authors reported a power calculation that was performed a 166 priori to estimate the sample size. Power calculations performed after completion of 167 the study were not considered. 168 Random sequence generation-Authors explicitly described the methods and 169 the type of randomisation to generate the random list. Two features were required to 170 be listed as "yes" [11]: (1) Explanation of the method by which the random sequence 171 was generated (i.e., computer, coin tosses, etc). A statement that a statistician performed the sequence generation was also valid. (2) Explanation of the type of 173 randomisation, e.g., simple randomisation, permuted block (to avoid imbalances in 174 allocation), stratification (to balance the distribution of certain baseline risk factors), 175 or a combination of these techniques. 176 Allocation concealment-The methods used to prevent the individuals 177 enrolling trial participants from knowing or predicting the allocation sequence in 178 advance (i.e., the method of preventing study personnel from having awareness of 179 treatment assignment before enrolling; [11]), were described in the article. Acceptable  Intention-to-treat-The article explicitly mentions that the analysis was made 194 on an "intention-to-treat" basis. groups) are provided with methods that estimate the effect size (i.e., risk ratio, odds 197 ratio, mean difference, etc.) with confidence interval, rather than solely "p values ". 198 199 For the purpose of the analyses, the domains "primary outcome", "power 200 calculation", "random sequence generation", "allocation concealment", and "use of 201 estimation methods" were considered always feasible, while the domains "blinding of 202 participants", "blinding of personnel", "blinding of outcome assessors" and 203 "intention-to-treat" were considered occasionally feasible, depending on study 204

characteristics. 205 206
Additional data extracted for each article 207 The following information was extracted from each article: Volume, Issue, 208 Title, Nationality of the affiliation of the first author, Objective or hypothesis of the 209 study. 210 211 Additional data extracted for each RCT 212 The following information was extracted from each RCT: 213 Number of participants -The total number of participants of each RCT was 214 extrapolated. If more than one RCT with the same treatments were described in the 215 same article, the total number of patients randomised was used. If the RCTs had 216 different treatments (i.e., one placebo-controlled, and one active-controlled) the 217 number of patients randomised in the placebo controlled RCT was employed. If a 218 RCT and a non-randomised study were described in the same article, only the number 219 of patients randomised was used. specific section of the article. The old definition of "randomised field trial" was also 246 considered acceptable. Randomised trials that were not registered in a trial repository 247 and did not use this terminology somewhere in the article were categorised as 248 "unstated RCT". A sub-category based on type and self-reconnaissance as RCT is: 249 "parallel manifest RCT", which is created to encompass all the randomised trial that 250 have the more classical design (parallel) and that are aware of being RCT. 251 Supplementary Table S1 297 Binary logistic regression outcome. Association between journal and prevalence of 298 RCTs. In this analysis the ORs represent the odds of publishing EoI studies with a 299 randomized controlled design, compared with JAVMA. Medical journals had from 6 300 to 15 times the odds of publishing a RCT compared with one of the veterinary 301