Evaluating the dose, indication and agreement with guidelines of antimicrobial use in companion animal practice with natural language processing

Abstract Background As antimicrobial prescribers, veterinarians contribute to the emergence of MDR pathogens. Antimicrobial stewardship programmes are an effective means of reducing the rate of development of antimicrobial resistance. A key component of antimicrobial stewardship programmes is selecting an appropriate antimicrobial agent for the presenting complaint and using an appropriate dose rate for an appropriate duration. Objectives To describe antimicrobial usage, including dose, for common indications for antimicrobial use in companion animal practice. Methods Natural language processing (NLP) techniques were applied to extract and analyse clinical records. Results A total of 343 668 records for dogs and 109 719 records for cats administered systemic antimicrobials from 1 January 2013 to 31 December 2017 were extracted from the database. The NLP algorithms extracted dose, duration of therapy and diagnosis completely for 133 046 (39%) of the records for dogs and 40 841 records for cats (37%). The remaining records were missing one or more of these elements in the clinical data. The most common reason for antimicrobial administration was skin disorders (n = 66 198, 25%) and traumatic injuries (n = 15 932, 19%) in dogs and cats, respectively. Dose was consistent with guideline recommendations in 73% of cases where complete clinical data were available. Conclusions Automated extraction using NLP methods is a powerful tool to evaluate large datasets and to enable veterinarians to describe the reasons that antimicrobials are administered. However, this can only be determined when the data presented in the clinical record are complete, which was not the case in most instances in this dataset. Most importantly, the dose administered varied and was often not consistent with guideline recommendations.


Introduction
Antimicrobial resistance (AMR) is an emergent global health crisis that was estimated to be responsible for the loss of over 700 000 lives in 2016, and this figure is estimated to grow to over 50 million by 2050 without intervention. 1 AMR reduces the therapeutic efficacy of antimicrobial treatment in both human and veterinary medicine with significant cost to patient health. As companion animals are able to acquire and exchange MDR pathogens with humans, and many of the same antimicrobial agents are used in human and veterinary medicine, companion animals can serve as a reservoir of AMR for in-contact people. [2][3][4][5] This highlights the importance of appropriate antimicrobial usage in companion animal practice and increases the imperative to adopt strategies to mitigate AMR in small animal veterinary clinics.
Antimicrobial stewardship programmes are a way to reduce AMR in hospital environments. [6][7][8][9][10] Measuring the effectiveness of antimicrobial stewardship programmes requires an understanding of the clinical indication for antimicrobial usage, as well as data on the antimicrobial usage patterns before and after the intervention. The clinical indication for antimicrobial use, the selection of an appropriate antimicrobial agent for that indication and administration of the correct dose for an appropriate duration of therapy are all key components of an appropriate use strategy that underpins good antimicrobial stewardship. The increasing number of practices with electronic health records, combined with data repositories such as VetCompass, 11 Small Animal Veterinary Surveillance Network (SAVSNET), 12 and Veterinary Medical Database (VMDB) 13 provide the opportunity to systematically collect these data. However, the data are primarily in the form of free text and not readily queryable, making data retrieval and subsequent analysis difficult. Previous studies have evaluated frequency and types of antimicrobial usage patterns in the UK 14,15 and Australia. 16,17 Studies describing the reason for antimicrobial administration have relied on the indication recorded when an appointment was booked, 18 or on the reason submitted in a claim to the pet insurance company. 19 However, neither of these studies examined the reason for the antimicrobial use directly from the veterinarian's clinical notes.
Extracting these data out of the clinical notes enables analysis of antimicrobial usage patterns directly based on the findings as recorded during the exam, and without interfering with the clinical workflow of the veterinarian. To perform such extraction and analysis at scale, we turn to automated methods based on natural language processing (NLP), which is a field of study that sits at the intersection of artificial intelligence and linguistics, with a broad goal of automating language analysis. 20 In our context of veterinary notes, NLP can be used to overcome the challenges of manual labelling of such data, enabling large-scale extraction of key antimicrobial usage information in a structured format, to allow subsequent analysis. 21 We focus on the extraction of actionable information from text, 22 and specifically on the use NLP for text mining, which is the discovery of non-trivial knowledge from unstructured text. 23 Models known as pre-trained contextualized language models have become popular in NLP, as they create a more nuanced and context-dependent representation of text than prior approaches. [24][25][26][27] These models capture the latent syntax and semantics of text, and support the training of task-specific models-such as identifying the reason for antimicrobial administration-more effectively with fewer labelled instances. Recent work in this area has integrated representations based on generalized texts, such as Wikipedia, with specialized texts, such as clinical notes, in order to more effectively analyse medical records. 28 One such model, 'VetBERT', 17 has been developed specifically for the veterinary domain was pre-trained using data from VetCompass Australia and further trained to classify the indication for which an antimicrobial was given out of free-text clinical notes.
The aim of this study was to characterize the reasons antimicrobials are given in veterinary practice, as described in the clinical notes by a consulting veterinarian. To this end, we apply VetBERT to the VetCompass Australia database. We also aim to determine details of the antimicrobial use, such as dose and length of administration, and assess the completeness of the electronic medical records to explore whether antimicrobial administration agrees with guidelines.

Materials and methods
De-identified clinical data from 137 companion animal practices was sourced from VetCompass Australia (Version 0.3) for the period 1 January 2013 to 31 December 2017 inclusive. 29 The antimicrobial agents administered and dosage information in each consultation were identified from free-text notes and collated using pre-existing NLP methods. 16,30,31 Where a unit is defined as a single capsule, tablet or millilitre of the antimicrobial, the dose extracted consists of the unit size, unit dose and frequency of administration. For example, one half 50 mg tablet given twice daily would be calculated as 50 mg for unit size, 0.5 for the dose unit and 2/day for the frequency. The patient weight and the total antimicrobial units dispensed were extracted from structured fields in the VetCompass record. The total daily dose of a medication was calculated by (unit size × dose unit × daily dose frequency)/ (weight of patient). The length of administration was calculated from (total number of units dispensed)/(total daily units). Where the item was injectable, the length of administration was fixed to one for the purposes of this report. A sample size of 97 is required to be 95% confident that estimated accuracy is within 10% of the actual range, based on Cochran's formula for the representativeness of proportions. 32 The accuracy of the automated dosage calculations created from the extracted data were assessed by randomly selecting 100 of the cases and manually calculating dosage.
A model referred to as VetBERT was used to classify the indication for disease. 17 VetBERT was created using a model known as ClinicalBERT 28 as a base and then using additional pretraining steps as described by Devlin et al. 24 using the entire corpus of 15 million clinical notes from VetCompass Australia to provide a good representation of the veterinary clinical text. One of 38 possible indications for antimicrobial use was obtained using VetBERT for each record (Table S1, available as Supplementary data at JAC-AMR Online). Further steps were taken to evaluate the performance of the model on records from our specific dataset. First, 50 records were randomly selected and labelled by three veterinarians to determine the inter annotator agreement of the veterinarians. Second, 100 records labelled by the model were labelled by a single veterinarian to determine the accuracy of the labels created by VetBERT.' Third, a further 400 records labelled by the model as 'no indication recorded' were randomly selected and split up, and each record was reviewed by one of three veterinarians to confirm the absence of a discernible indication.
Code was written in Python, and machine learning and statistical tests on algorithms performed with scikit-learn and TensorFlow libraries. All descriptive statistics, computations and visualizations were performed using Tableau 2020. 33 Doses of antimicrobials were grouped by rounding to the nearest integer and given a tolerance of 10% before being compared with the Australian Veterinary Prescribing Guidelines from The University of Melbourne. 34 Statistical significance was tested using Pearson's χ 2 test with P , 0.001 used to indicate significance.

Results
Clinical data relating to 4 402 147 consultation records from 137 companion animal practices were aggregated for analysis (Table 1). Clinical records from 3 269 160 consultations for 513 962 dogs and 1 132 987 consultations for 199 358 cats were included in the analysis. Occasionally, a dog and cat were recorded within the same consultation. Of the consultation records analysed, 199 358 (26%) were cats and 513 962 (74%) dogs. Systemic antimicrobials were administered or dispensed in 109 719 (9.7%) of the cat consultations and 343 668 (11%) of dog consultations.
The manual dosage calculations exactly matched the calculations from the extracted data in 94% of cases. The Fleiss Kappa agreement score between the veterinarians was excellent (0.77) in determining the reason for an antimicrobial administration. The model had an accuracy of 82% in identifying the indication Hur et al.
for an antimicrobial if one was dispensed, and accuracy of 81% when the model labelled 'no indication recorded'. Of the records that were labelled with 'no indication' 80% were confirmed to have no identifiable indication.
For all consultations where systemic antimicrobials were administered, dispensed or prescribed the dose, duration and indication for antimicrobial usage were extracted. Of the 453 387 consultations involving dogs and cats when antimicrobials were administered, only 173 887 (38%) recorded the indication, dose and duration ( Figure 1). The remaining consultation records were lacking one or more of these variables, with 67 528 (15%) containing no discernible indication, 265 513 (59%) recording no dose rate, 30 932 (7%) with no recorded duration of treatment and 9923 (2.2%) of consultation records containing no indication, dose or duration data that could be extracted by the algorithms (Figure 1).

Guideline comparison
The daily doses of active ingredient (mg/kg) were calculated from the clinical records for the most commonly used antimicrobial agents, amoxicillin/clavulanate, cefalexin, enrofloxacin, metronidazole and cefovecin, for both cats and dogs (Figure 4). The reported daily dose in the consultation record for these medications was compared with the dose range recommended in the Australian Veterinary Prescribing Guidelines. The daily dose of administration for cefovecin had the highest proportion of doses within the guidelines, which was 95% for dogs ( Table 2) and  Antimicrobial indication and dosing rates in companion animals Hur et al. Antimicrobial indication and dosing rates in companion animals 93% for cats (Table 3). Doxycycline had the lowest proportion of administered doses that were compliant with the guidelines for dogs (35%) ( Table 2), and enrofloxacin the lowest for cats (25%) ( Table 3).

Discussion
In this study, we applied NLP techniques to evaluate a large dataset of veterinary notes to determine the reason for, and     35,36 Evaluating agreement with guidelines in Australian hospitals has been fundamental in driving antimicrobial stewardship interventions. 37 General medical practices are similar to veterinary practices in that they are often run by a single practitioner and cannot have dedicated staff to perform antimicrobial stewardship activities. The previous studies evaluating general medical practices used post hoc analysis, which used a field to specify the reason for the antimicrobial being dispensed. However, it was found that over 85% of these labels did not have the reason recorded, and it is likely the clinical notes themselves need to be examined to determine this. 38 NLP methods, such as the ones demonstrated in our study, could potentially benefit these programmes by retrieving this information from the clinical notes themselves. Similarly, having an indication for an antimicrobial being dispensed could simplify the extraction of the indication of use. Further research is required to evaluate the utility of NLP techniques for investigating appropriateness in general medical practice.
The most common indication for antimicrobial administration in dogs was skin disorders and in cats it was traumatic injuries. This was similar to the findings from a study evaluating pet insurance claims in Australia 19 and a study evaluating patients in the UK. 14 Despite being the most common reason for antimicrobial administration, only 31% of the consultation records for dogs with a skin disorder were treated with systemic antimicrobials (Figure 2a). This would appear to be consistent with guidelines, which only recommend systemic antimicrobials when there are large areas of the body involved. The choice of amoxicillin/clavulanate for traumatic injuries in cats and dogs also appears to be in concordance with guidelines, although amoxicillin alone is recommended for abscesses and traumatic injuries rather than amoxicillin/clavulanate. 34 However, further research is required to understand the clinical severity of these conditions and confirm the appropriateness of the chosen therapy.
While there are limitations of this study, it is a broad assessment of the antimicrobial selection for disease conditions and the dose of therapy chosen by the practitioner. Antimicrobial agent selection was broadly concordant with guideline recommendations in most common scenarios with some exceptions, such as traumatic injuries in cats. However, dose rate varied widely and may potentially be inappropriate (27%) when compared with the Australian Veterinary Prescribing Guidelines ( Table 2). Underdosing of antimicrobials results in failure to achieve adequate MIC, which promotes AMR. 39 This may be associated with a lower than expected number of records where the patient weight was recorded (60%), but may also be associated with the widely variable weight of the patients (Pomeranian versus Great Dane) and the lack of appropriately formulated dosages of common antimicrobials. A portion of the incorrect dosing is likely due to discrepancies between the label dose and available best-practice recommendations, as has been previously described in Australia. 40 For example, while the available literature supports the dose rate of doxycycline being 5 mg/kg given twice daily, 41 at least one label for doxycycline lists the dosage rate to be only 2.5 mg/kg for a daily dose. 42 Similarly, at least one cefalexin label recommends 15 mg/kg every 12 h, 43 where studies suggest that anything below 25 mg/kg should be given every 6-8 h to achieve the MIC necessary to be effective. 44 Also, when no range is provided by guidelines, this can impact the proportion of cases where the appropriate dose is used. For example, doxycycline has a daily dose of 10 mg/kg; however, the combination of animal weight and tablet size may make it difficult to achieve a precise dose in many cases. Research is needed to investigate these factors.
Natural language processing is a powerful tool for automated data extraction and large-scale monitoring of antimicrobial administration on a large population level without the need for manual record labelling, as done in previous studies. 11,18,45 However, these methods are limited by the completeness of the information recorded in the clinical notes. In addition, evaluating appropriate antimicrobial use requires the clinical record to clearly state the indication, the agent selected and the dose and duration of use. In this study, only 38% of the records analysed had all the information necessary to determine whether the antimicrobial usage was appropriate. The NLP algorithms were accurate, with a 94% accuracy for the dosing calculations correct and 80% accuracy in identifying the indication of the antimicrobial. The primary components of data missing included the weight of the animal, unit measurements for the antimicrobial agent, prescription details or a clear reason for the antimicrobial administration being described. While there is likely an overestimate of the doses missing from the records, the information necessary to perform this calculation, using the prescription or the concentration of the medication given, was not easily available. Missing data in medical records has also been reported in human health; a study investigating human primary care visits reported at least 14% of records were missing information that could adversely affect the clinical interpretation of the record. 46 Further research is required to understand the reason for the missing data in many of the records, as it is not clear whether the data are not being entered at the time of the consultation, are entered in the wrong place or were lost during export of the records from the practices. The importance of having a complete clinical record is highlighted by the proportion of cats that received cefovecin, a highly critically important third-generation cephalosporin, when no indication for treatment was recorded. While giving a long-acting antimicrobial such as cefovecin may be appropriate, the reason for administration should be clearly indicated.
The analysis in this study was limited to VetCompass 0.3, and the methods had the same limitations previously described. 29 The diseases were extracted at the syndrome level and specific clinical details of the syndromes were not extracted in enough detail to make an evaluation of appropriateness. This would require a detailed exploration of each disease syndrome and understanding what information is possible to extract from the clinical record. Additionally, the indication for the disease was counted on a per consultation record basis, and a patient seen multiple times and hospitalized may have multiple records for the same indication. While there were repeat visits labelled (11% in dogs and 15% in cats), these visits were not associated with the condition for which it was a repeat appointment and relied on text indicating a repeat visit. So, if there are two Antimicrobial indication and dosing rates in companion animals consultation records for the same patient with an abscess, and only one appointment with an antimicrobial administered, this could potentially be counted as one consultation for a patient that had an abscess and received an antimicrobial, and one consultation where a patient had an abscess and did not receive an antimicrobial. Further research would be required to understand the impact on each condition. This also made it unreasonable to evaluate the appropriateness of the duration in this study as prescription repeats and chronic conditions (i.e. skin conditions in dogs), with multiple consultations spread out over time, require additional methods to accurately assess duration of therapy.
There was also a lack of culture and susceptibility testing results in the data analysed. Increased use of bacterial cultures and susceptibility testing is likely to improve the clinical outcomes while reducing the importance ratings of the antimicrobials used. While there were labels with culture and sensitivity annotated in the original corpus used to train the disease syndrome classifier, 17,47 there were not enough of these labels to train the model. Evaluation of the antimicrobial susceptibility of isolates obtained from canine urine cultures in Australia demonstrated that antimicrobial agents of lower critical importance could be selected without compromising efficacy, 48 but further work is required to incorporate clinical pathology data into VetCompass and evaluate this in a large scale dataset.

Conclusions
Utilizing natural language processing and records from VetCompass Australia, we have performed a large-scale analysis of the indication and dose of antimicrobial use in companion animal practice in Australia. We have demonstrated the utility of automated methods to support understanding why and how antimicrobials are being administered and describe how they agree with antimicrobial guidelines in instances where the data are able to be extracted. Further research is required to understand other factors relating to these behaviours and details of outcomes of treatments.