Results of the 2020 Genomic Proficiency Test for the network of European Union Reference Laboratory for Antimicrobial Resistance assessing whole-genome-sequencing capacities

The global surveillance and outbreak investigation of antimicrobial resistance (AMR) is amidst a paradigm shift from traditional biology to bioinformatics. This is due to developments in whole-genome-sequencing (WGS) technologies, bioinformatics tools, and reduced costs. The increased use of WGS is accompanied by challenges such as standardization, quality control (QC), and data sharing. Thus, there is global need for inter-laboratory WGS proficiency test (PT) schemes to evaluate laboratories' capacity to produce reliable genomic data. Here, we present the results of the first iteration of the Genomic PT (GPT) organized by the Global Capacity Building Group at the Technical University of Denmark in 2020. Participating laboratories sequenced two isolates and corresponding DNA of Salmonella enterica , Escherichia coli and Campylobacter coli , using WGS methodologies routinely employed at their laboratories. The participants' ability to obtain consistently good-quality WGS data was assessed based on several QC WGS metrics. A total of 21 laboratories from 21 European countries submitted WGS and meta-data. Most delivered high-quality sequence data with only two laboratories identified as overall underperforming. The QC metrics, N50 and number of contigs, were identified as good indicators for high-sequencing quality. We propose QC thresholds for N50 greater than 20 000 and 25 000 for Campylobacter coli and Escherichia coli, respectively, and number of contigs >200 bp greater than 225, 265 and 100 for Salmonella enterica , Escherichia coli and Campylobacter coli , respectively. The GPT2020 results confirm the importance of systematic QC procedures, ensuring the submission of reliable WGS data for surveillance and outbreak investigation to meet the requirements of the paradigm shift in methodology.


Participating laboratories
Participation was offered to the European network of National Reference Laboratories (NRL) for antimicrobial resistance within the food and veterinary sectors.Invitation letters were distributed on the 20 th of December 2019 by e-mail to the EURL-AR network to announce participation in the GPT20.Participating laboratories are identified by user codes and are confidential.Participant identity is only known by the corresponding laboratory and the GPT organizers, including the European Commission (EC) and the European Food Safety Authority (EFSA).

Reference strains
Two isolates of three bacterial species were selected for the GPT20, among strains with resistance mechanisms or profiles of particular interest to the EURL-AR network.The selected species for the GPT20 were the following priority pathogens: Salmonella enterica (S. enterica), Escherichia coli (E.coli), and Campylobacter spp.. Genomic DNA was extracted from the six GPT isolates using an Invitrogen Easy-DNA KitTM (Invitrogen, Carlsbad, CA, United States) and DNA concentrations were determined using the Qubit dsDNA BR assay kit (Invitrogen).The genomic DNA of each strain was divided into two fractions of which one was dried and supplemented with DNA stabilizing agent (DNAstable® Plus, Biomatrica, https://www.biomatrica.com/download/hp-dnastable-plus-handbook/)and sent to the participating laboratories as GPT material as well as being used to provide a closed reference genome.The remaining fraction was prepared for Illumina pair-end sequencing using the Illumina (Illumina, Inc., San Diego, CA, United States) Nextera XT® Guide 150319425031942 following the protocol revision C1.A sample of the pooled Nextera XT Libraries was loaded onto an Illumina NextSeq reagent cartridge using NextSeq Reagent Kit v2.The libraries were sequenced using an Illumina NextSeq platform.
To enable the assessment of the differences in the sequences generated by the laboratories, for each of the GPT strains closed genomes were produced.Multiplexed microbial SMRTbell libraries were prepared using the SMRTbell prep kit 3.0 according to PacBio protocol (PacBio, Menlo Park, CA, November 2021).The multiplexed SMRTbell library was then sequenced on the Sequel IIe system (PacBio, Menlo Park, CA) using Binding Kit 3.2 and Sequel sequencing kit 2.0 on one SMRT cell 8M (PacBio, Menlo Park, CA), with 30 hours collection time.Raw HiFi reads were demultiplexed by running the Demultiplex Barcodes application and de novo assembled using the Microbial Assembly in SMRTLink (PacBio, Menlo Park, CA).The resulting assemblies were circularized and oriented to start at oriC. .The closed reference genomes were analyzed using in silico bioinformatics tools to confirm the identify the MLST (Multilocus sequence typing) sequence type (ST) genes using the pipeline; MLST v 2.0.4 available from Center for Genomic Epidemiology http://www.genomicepidemiology.org/

Distribution of reference material
Individual sets of the reference material were distributed by the EURL AR to each participating laboratory on the 22 nd of September 2020, along with a cover letter.Each participant received two isolates of each species named (GPT20-001-6 (BACT)), as well as corresponding genomic DNA named (GPT20-001-6 (DNA)).The reference material was distributed in double pack containers (class UN 6.2) according to the International Air Transport Association (IATA) regulations as UN3373, biological substances Category B.

Procedure
Protocols on how to handle the received bacterial cultures and DNA were made available on the Global Surveillance website (https://www.globalsurveillance.eu/projects/genomic-proficiencytest-2020).Additional relevant information was distributed by email directly to the Participating laboratories were asked to perform the GPT2020 using WGS methodologies routinely employed at their laboratories.
Furthermore, the participants were requested to capture relevant information via an online survey, which contained questions about the methodology applied.This included information such as the cultivation and DNA extraction procedure, the quality assurance metrics applied, details related to the sequencing and analysis of the obtained sequencing data (Supplementary Table 1)

Sequencing quality analysis
The laboratories were requested to submit their generated WGS raw sequence files in fastq format to a DTU hosted ftp-site.Subsequently, the raw reads were also submitted by the GPT provider to the European Nucleotide Archive (ENA) under project accession number PRJEB58706.
As part of the GPT, these reads were run through an in-house pipeline, where the reads were de novo assembled using the SPAdes v 3.9.0software (Prjibelski et al., 2020).Subsequently, reads were aligned to closed reference genome using Burrows-Wheeler Aligner (BWA)-MEM algorithm v 0.7.12 (Li, 2013) with default settings and Samtools v 1.2 (Li et al., 2009) was used to filter the reads that did not map.Finally, each of the QC metrics listed in Table 2 were calculated.The statistical analysis of each calculated QC metric as well as outliers were visualized in twelve boxplots (Supplementary Figure 1).
To further support the assessment of the participating laboratories´ sequencing quality, the MLST of each isolate was predicted using a command line version of the CGE MLST v 2.0.4 and newest database version available (March 10 th , 2021) (Larsen et al., 2012) in addition to a phylogenetic analysis.In specific, single nucleotide polymorphism (SNP) matrices were created for each of the six organisms by using the raw reads of both the culture and corresponding purified DNA submitted by the laboratories.SNPs were determined using the CGE pipeline CSI phylogeny v.1.4available at (https://cge.cbs.dtu.dk/services/CSIPhylogeny/)(Kaas et al., 2014).The reads were mapped to the reference genomes; using BWA version 0.7.13 2013), producing a matrix of SNP differences.

Correlation statistics
The metrics "N50" and "contigs with more than 200 bp" have previously been regarded as the most suitable for estimating sequencing quality (Earl et al., 2011).Here, a linear regression analysis was performed with the purpose of elucidating whether these correlated with other quality metrics and identify those best associated with sequencing quality.Initially, data was cleaned in two steps to ensure trustworthy results, by removing samples larger than 105% or smaller than 95% of assembly size compared to the closed reference genomes or with incorrect or incomplete MLST-scheme genes identified.Subsequently, data was cleaned for outliers by removing isolates which deviated by more than 3SD from the mean for each sample type.Outlier detection was based on the following metrics: average insert size, quality score of the forward and reverse read separately, total number of contigs, number of contigs more than 200 bp, proportion of reads mapped to the reference genome, N50 and NG50.This restriction was applied as these metrics describe overall quality, while excluded metrics were tied to the yield of the sequencing.Following the exclusion of poor quality data, a linear regression analysis was performed including all the QC parameters found in table 2, using the linear model (lm) in R. Accepted correlation had a p-value less than 0.05 for half (5 out of 10) sample types and Pearson correlations were calculated using the in-built R function (cor).The correlations were visualized using the R-libraries reshape2 and ggplot2 (figure 1).
The SD and mean were recalculated for the cleaned dataset, and a threshold of 3SD below/above the mean, depending on the specific QC-metric (referred to as adjusted quality threshold below) was set to determine laboratories who performed considerably less well comparable to other participants and to define minimum QC-thresholds for acceptable sequence quality.The minimal QC-thresholds for acceptable sequence data was suggested by taking the average of the obtained 3SD levels for each species separately, calculated from the filtered data and excluding laboratories identified as general underperformers.

General underperformance
To identify the laboratories that underperformed overall and need to assess their workflow, a subset of the QC metrics was selected based on the correlation statistics.Values below the adjusted quality threshold in one QC metric, would increase the risk of failing those correlated to it.If a participant was below the adjusted quality threshold across uncorrelated QC metrics, the laboratory is more likely to suffer from issues in their workflow.Failure across these uncorrelated metrics was termed general laboratory underperformance.