Benchmarking Long-Read Assemblers for Genomic Analyses of Bacterial Pathogens Using Oxford Nanopore Sequencing

Oxford Nanopore sequencing can be used to achieve complete bacterial genomes. However, the error rates of Oxford Nanopore long reads are greater compared to Illumina short reads. Long-read assemblers using a variety of assembly algorithms have been developed to overcome this deficiency, which have not been benchmarked for genomic analyses of bacterial pathogens using Oxford Nanopore long reads. In this study, long-read assemblers, namely Canu, Flye, Miniasm/Racon, Raven, Redbean, and Shasta, were thus benchmarked using Oxford Nanopore long reads of bacterial pathogens. Ten species were tested for mediocre- and low-quality simulated reads, and 10 species were tested for real reads. Raven was the most robust assembler, obtaining complete and accurate genomes. All Miniasm/Racon and Raven assemblies of mediocre-quality reads provided accurate antimicrobial resistance (AMR) profiles, while the Raven assembly of Klebsiella variicola with low-quality reads was the only assembly with an accurate AMR profile among all assemblers and species. All assemblers functioned well for predicting virulence genes using mediocre-quality and real reads, whereas only the Raven assemblies of low-quality reads had accurate numbers of virulence genes. Regarding multilocus sequence typing (MLST), Miniasm/Racon was the most effective assembler for mediocre-quality reads, while only the Raven assemblies of Escherichia coli O157:H7 and K. variicola with low-quality reads showed positive MLST results. Miniasm/Racon and Raven were the best performers for MLST using real reads. The Miniasm/Racon and Raven assemblies showed accurate phylogenetic inference. For the pan-genome analyses, Raven was the strongest assembler for simulated reads, whereas Miniasm/Racon and Raven performed the best for real reads. Overall, the most robust and accurate assembler was Raven, closely followed by Miniasm/Racon.


Introduction
The rapid development of whole-genome sequencing (WGS) technologies over the last decade has revolutionized the ability to monitor and trace outbreaks of infectious diseases [1]. Illumina short-read sequencing has been widely used for sequencing bacterial pathogens, which can produce millions of short reads (<300 bp) in length and low error rates of 1-2% [2]. A high-quality bacterial genome assembly should be both accurate and complete based on the concept of "one chromosome, one contig" proposed by Koren and Phillippy [3]. However, Illumina sequencing is generally insufficient for assembling complete genomic structures to investigate how genes are organized within genomes and whether these genes are located on the chromosome or plasmids of bacteria [4]. This limitation is mainly attributed to the fact that short reads are not In the present study, we assessed the aforementioned long-read assemblers in the context of Oxford Nanopore sequencing of major bacterial pathogens. The selected species covered a wide range of genome sizes and GC contents. We evaluated and compared the capabilities of these assemblers in generating complete and accurate assemblies in terms of genome completeness and accuracy, antimicrobial resistance (AMR), virulence potential, phylogeny, and pan genome. Specifically, we utilized both simulated (mediocre-and low-quality) and real reads for selected species of bacterial pathogens. Simulated reads allow for confident ground truth, as the underlying reference genome is known with certainty. Alternatively, the use of real reads can better inform us of how real reads interact with the assembly algorithms to produce accurate genomic assemblies. By taking into consideration both types of data on species with different genomic characteristics, we can better evaluate the underlying algorithms of each assembler, and how they may affect the completeness and accuracy of genome assemblies and downstream genomic analyses.

Genome Completeness and Accuracy
For simulated reads which contained artificial error profiles, genome completeness and accuracy indicate the robustness of an assembler to tolerate a wide range of read parameters such as chimeras, low-quality regions, and systematic basecalling errors [17]. The Canu, Miniasm/Racon, and Raven assemblies of mediocre-quality reads had numbers of contigs not significantly different (p > 0.05) from the reference genomes (Table 1), while Flye, Redbean, and Shasta produced significantly higher (p < 0.05) numbers of contigs than the reference genomes. The Redbean assemblies generated genome sizes significantly larger than the reference genomes (p < 0.05), whereas other assemblers produced assemblies with significantly lower (p < 0.05) genome sizes than the reference genomes. The GC contents of all assemblies were not significantly different (p > 0.05) from those of the reference genomes. The complete benchmarking universal single-copy orthologs (BUSCOs) of all assemblies were significantly lower (p < 0.05) than those of the reference genomes. Significantly higher (p < 0.05) complete BUSCOs were observed in the Miniasm/Racon and Raven assemblies compared to other assemblers (Supplementary Table S1), while the complete BUSCOs of the Raven assemblies were significantly higher (p < 0.05) than those of the Miniasm/Racon assemblies. Significantly lower (p < 0.05) numbers of SNPs and indels were detected in the Canu, Miniasm/Racon, and Raven assemblies than other assemblies (Supplementary Table S2). The numbers of SNPs and indels of the Canu and Raven assemblies were not significantly different (p > 0.05) but were significantly lower (p < 0.05) than the Miniasm/Racon assemblies.
We also tested simulated Oxford Nanopore long reads of low quality to examine whether each long-read assembler could tolerate a higher degree of errors. When assembling low-quality reads, Canu and Shasta failed to generate assemblies for all strains (Table 2), while Flye and Redbean did not produce assemblies for three and one out of 10 strains, respectively. Similarly, Wick and Holt (2020) also reported the incomplete assembly processes of Canu, Flye, Redbean, and Shasta when simulated reads of prokaryote genomes were tested. In contrast, Miniasm/Racon and Raven generated assemblies for all strains. Compared to mediocre-quality reads, Flye, Miniasm/Racon, Redbean, and Raven produced assemblies with more fragmented contigs using low-quality reads. The numbers of contigs of the Raven and Redbean assemblies were not significantly different (p > 0.05) from those of the reference genomes, whereas Flye and Miniasm/Racon generated significantly more (p < 0.05) contigs than the reference genomes. The genome sizes and GC contents of the Raven assemblies were still not significantly different (p > 0.05) from those of the reference genomes, although low-quality reads were used. Flye, Miniasm/Racon, and Redbean all produced significantly different (p < 0.05) GC contents from the reference genomes. The complete BUSCOs of the Raven assemblies were still significantly higher (p < 0.05) than those of the Flye, Miniasm/Racon, and Redbean assemblies (Supplementary Table S3). As Flye and Miniasm/Racon produced assemblies with inaccurate genome sizes using low-quality reads, the numbers of SNPs and indels of the assemblies of low-quality reads were not compared with those of mediocre-quality reads. Significant increases (p < 0.05) in the numbers of SNPs and indels were observed in the Raven assemblies of low-quality reads compared to mediocre-quality reads (Supplementary Table S4).  Compared to when mediocre-quality reads were tested, the numbers of contigs of the Flye, Miniasm/Racon, Raven, and Redbean assemblies of low-quality reads became significantly higher (p < 0.05) when using low-quality reads. No significant differences (p > 0.05) were observed between the genome sizes of the Raven assemblies of mediocre-and low-quality reads, while the genome sizes of the Flye, Miniasm/Racon, and Redbean assemblies of low-quality reads were significantly lower (p < 0.05) than those of mediocre-quality reads. There were no significant differences (p > 0.05) in GC content between the Flye, Miniasm/Racon, Raven, and Redbean assemblies of mediocre-and low-quality reads. The complete BUSCOs of the Flye, Miniasm/Racon, Raven, and Redbean assemblies of low-quality reads were significantly lower than those of mediocre-quality reads (p < 0.05), while significantly higher (p < 0.05) numbers of SNPs and indels were observed when low-quality reads were tested.
Despite the advantages of simulated reads in allowing for a controlled evaluation of the effect of long-read quality on genome assembly, they cannot completely capture the challenges of using real Oxford Nanopore reads when assessing long-read assemblers. The genome completeness and accuracy of an assembler indicate its reliability to achieve a complete and accurate assembly given a set of real reads, which incorporate naturally occurring features of Oxford Nanopore long reads (e.g., error profiles, read lengths, quality scores) [17]. All assemblers completed assembly processes for each strain to generate assemblies when real reads were used (Table 3). Canu, Flye, and Raven were superior to other assemblers, and produced the most contiguous assemblies, while other assemblers produced inaccurate numbers of contigs for some strains. For example, Miniasm/Racon produced 227 and 43 contigs for Clostridium botulinum CFSAN034200 and C. jejuni NCTC 11168, respectively. Shasta generated 850 contigs for C. sakazakii CFSAN068773. Redbean performed poorly in achieving contiguous assemblies, often producing inaccurate numbers of contigs. The genome sizes and GC contents of the Miniasm/Racon, Raven, Redbean, and Shasta assemblies were not significantly different (p > 0.05) from those of the reference genomes, whereas Canu and Flye failed to produce assemblies which had genome sizes and GC contents not significantly different from the reference genomes (p > 0.05). The complete BUSCOs of all assemblies were significantly lower (p < 0.05) than those of the reference genomes (Supplementary Table S5). The Miniasm/Racon and Raven assemblies had the highest (p < 0.05) complete BUSCOs compared to other assemblers, while no significant differences (p > 0.05) were observed between the Miniasm/Racon and Raven assemblies. The numbers of SNPs of the Canu assemblies were significantly lower (p < 0.05) than those of the assemblies produced by other assemblers (Supplementary Table S6), while no significant differences (p > 0.05) were found among other assemblers, with only one exception that the Raven assemblies had the numbers of SNPs significantly lower (p < 0.05) than the Miniasm/Racon assemblies. The Raven assemblies had the lowest (p < 0.05) numbers of indels among all assemblers. As predicted with the aid of the PlasmidFinder database, the assemblies of simulated reads showed inconsistent plasmid profiles (types and numbers of plasmids) with the reference genomes in some cases (Supplementary Tables S7 and S8). All assemblies of Bacillus anthracis Ames Ancestor showed consistent plasmid profile with the reference genome. Only the Flye assembly of S. Typhimurium LT2 contained both IncFIB (S) and IncFII (S), which was consistent with the reference genome. The plasmid profiles of the Redbean and Shasta assemblies of Cronobacter sakazakii ATCC 29544 were consistent with that of the reference genome. None of the assemblies of Escherichia coli O157:H7 Sakai carried IncFIB (AP001918) that was present in the reference genome. Among the assemblies of low-quality reads, the Raven assembly of Staphylococcus aureus TW20 was the only one that carried a plasmid, whereas plasmids were not present in other assemblies.
Similarly, plasmid profiles inconsistent with the reference genomes were also observed in the assemblies of real reads (Supplementary Table S9). Compared to the reference genomes, the Shasta assembly of S. Bareilly CFSAN000189 did not harbor IncFII (S). The Flye assembly of S. aureus CFSAN007894 had a consistent plasmid profile with the reference genome, while inconsistent plasmid profiles were identified in other assemblies. Interestingly, similar to Escherichia coli O157:H7 Sakai with simulated reads, none of the assemblies of E. coli O157:H7 CFSAN076619 possessed IncFIB (AP001918) that was contained in the reference genome, which could be attributed to the loss of this plasmid during Oxford Nanopore library preparation. This finding also pointed to the suboptimal library preparation bias as a second possible explanation [18]. Consequently, there is a current need for reliable protocols to extract high-molecular-weight genomic DNA from bacteria that are compatible with Oxford Nanopore sequencing to overcome this deficiency. The degree to which Oxford Nanopore sequencing is compatible with diverse DNA extraction chemistries remains to be validated.
According to the assemblies using both simulated and real reads, overall, Raven emerged as the best assembler that can be tolerant of both low-identity and realistic reads. Noticeably, Raven was the most robust assembler that was not significantly affected by random reads, junk reads, or chimeric reads in low-quality reads. It should be noted that one expected benefit of Oxford Nanopore sequencing is that it can identify the exact locations of mobile genetic elements (MGEs) such as plasmids. Unfortunately, since Raven does not indicate the circularity of contigs that harbor plasmids, it is not able to determine whether plasmids are integrated into the chromosome or exist extra-chromosomally in a non-integrated state. This drawback of Raven may potentially cause an imperfect representation of the genetic architectures of MGEs.

Antimicrobial Resistance Genes (ARGs)
Oxford Nanopore sequencing has great potential for accelerating AMR genotyping, as sequence data are acquired in real time within minutes of sequencing [19,20]. Meanwhile, Oxford Nanopore sequencing enables the identification of MGEs on which ARGs are located and also characterizes the combination of different ARGs co-located on the same MGEs. However, it is worth noting that Oxford Nanopore sequencing currently has higher error rates than Illumina sequencing, although improvements in nanopore chemistry and basecalling could potentially reduce the differential [21]. The error rate of Oxford Nanopore long reads could also in part be compensated for by assembly algorithms through intensive computations to acquire a more accurate AMR profiling independent of other sequencing technologies such as Illumina sequencing. Therefore, we compared the AMR genotypes and phenotypes of selected bacterial pathogens, as predicted based on Oxford Nanopore long-read assemblies using different long-read assemblers.
Five genotypically antimicrobial-resistant strains with simulated reads were used (Supplementary Table  S10). All Miniasm/Racon and Raven assemblies provided AMR profiles that were consistent with the reference genomes. Canu also produced assemblies that had accurate AMR profiles, with only one exception that the Canu assembly of S. aureus TW20 carried erm(33) that was absent in the reference genome. Both Flye and Shasta produced assemblies with inaccurate AMR profiles for three strains, while Redbean performed the worst and produced four Redbean assemblies with inconsistent AMR profiles with the reference genomes. Noticeably, when low-quality reads were used, no ARGs were identified in the Flye, Miniasm/Racon, and Redbean assemblies (Supplementary Table S11). Raven still produced the assembly of Klebsiella variicola DSM 15968 that had an accurate AMR profile, while the Raven assemblies of both P. aeruginosa PAO1 and C. jejuni NCTC 11168 also contained three ARGs that were present in the reference genomes.
Five genotypically antimicrobial-resistant strains with real reads were used (Supplementary Table S12). All assemblies of P. aeruginosa CFSAN084950 had consistent AMR profiles with the reference genome. The gene associated with kanamycin resistance, aph(3 )-III, was present in the Canu, Miniasm/Racon, Raven, and Redbean assemblies of E. coli O157:H7 CFSAN076619, while the gene associated with ampicillin resistance, blaOXA-61, was only identified in the Miniasm/Racon and Raven assemblies of C. jejuni NCTC 11168. Noticeably, none of the assemblies of S. aureus CFSAN007894 and Campylobacter coli CFSAN032805 carried the same ARGs with the reference genomes.
Overall, compared to other assemblers, Raven was more capable of generating assemblies for accurate AMR genotypes, closely followed by Miniasm/Racon, especially when low-quality reads were used. Although it is possible to assemble Oxford Nanopore long reads into complete genomes to help implement near real-time AMR profiling, doing so would compromise the genome accuracy of bacterial pathogens and result in inaccurate AMR genotypes. Future improvements to the library preparation and basecalling of Oxford Nanopore sequencing, as well as long-read assembly algorithms, may mitigate this deficiency.

Virulence Genes
Previous studies have demonstrated the effectiveness of Oxford Nanopore sequencing in the rapid identification of virulence genes in bacterial pathogens [22][23][24]. The numbers of virulence genes identified in all assemblies of mediocre-quality reads of each strain were not significantly different (p > 0.05) from those in the reference genome (Table 4). However, it should be noted that when low-quality reads were tested, only the Raven assemblies had numbers of virulence genes that were not significantly different (p > 0.05) from the reference genomes (Table 5). No significant differences (p > 0.05) were observed between the numbers of virulence genes of the Raven assemblies of mediocre-and low-quality reads, while the numbers of virulence genes of the Flye, Miniasm/Racon, and Redbean assemblies of low-quality reads became significantly lower (p < 0.05) compared to those of mediocre-quality reads. All assemblies of real reads of each strain had numbers of virulence genes not significantly different (p > 0.05) from those in the reference genome (Table 6). Taken together, all assemblers were able to generate assemblies containing accurate information about the virulence potential of bacterial pathogens, when Oxford Nanopore long reads with a lower level of errors were assembled. However, when more errors were introduced into Oxford Nanopore long reads, Raven was superior to other assemblers.

Multilocus Sequence Typing (MLST)
MLST has been considered as a portable approach to the identification of clones within populations of pathogens [25], thus enabling an exchange of molecular typing data for global epidemiology via the Internet. Tarumoto et al. [26] reported that the Canu assemblies of the Oxford Nanopore long reads of two Enterococcus faecium strains provided consistent MLST results with conventional Sanger sequencing, although a few mismatched bases on each locus were observed, especially in repeat sequences. In this study, we thus investigated which assemblers could best assemble Oxford Nanopore long reads for accurate MLST of bacterial pathogens.
It is worth noting that Miniasm/Racon was the most effective assembler for mediocre-quality reads (Table 7), whose assemblies contained the seven housekeeping genes for all strains tested. Canu and Raven also performed well for MLST, as only the Canu and Raven assemblies of C. botulinum CDC_1632 failed to have a positive MLST result. The Flye assemblies showed positive MLST results only for three strains, while the Redbean assemblies of up to eight strains had negative MLST results. Shasta performed the worst and did not generate any assemblies with positive MLST results. Noticeably, when the assemblies of low-quality reads were evaluated for MLST (Table 8), the Raven assemblies of E. coli O157:H7 Sakai and K. variicola DSM 15968 showed positive MLST results among all strains tested. Table 4. Numbers of virulence genes in bacterial strains with mediocre-quality reads compared to their corresponding reference genomes, as predicted based on their Oxford Nanopore long-read assemblies using different long-read assemblers.  Table 5. Numbers of virulence genes in bacterial strains with low-quality reads compared to their corresponding reference genomes, as predicted based on their Oxford Nanopore long-read assemblies using different long-read assemblers.    Table 7. Multilocus sequence typing (MLST) of bacterial strains with mediocre-quality reads compared to their corresponding reference genomes, as predicted based on their Oxford Nanopore long-read assemblies using different long-read assemblers.

Campylobacter jejuni NCTC 11168
Canu    Canu Concerning real reads, Miniasm/Racon and Raven performed equally well (Table 9), as their assemblies did not harbor all seven housekeeping genes only for C. botulinum CFSAN034200. Canu and Redbean produced assemblies of C. coli CFSAN032805 and C. jejuni NCTC 11168 with negative MLST results. In contrast, there were six Shasta assemblies with positive MLST results, while only the Flye assemblies of P. aeruginosa CFSAN084950 and C. sakazakii CFSAN068773 had positive MLST results.

Whole-Genome Phylogeny
Taylor et al. [27] used 23 closely related Salmonella strains (number of SNPs < 500) to build a whole-genome phylogenetic tree with the Miniasm/Racon assembly of the Oxford Nanopore long reads of S. Bareilly CFSAN000189, where they observed a congruent topology between trees built with the SPAdes assembly of Illumina short reads and the Miniasm/Racon assembly. In our work, both closely and distantly related strains were included when evaluating the performance of Oxford Nanopore long-read assemblies in the phylogenetic analyses. A total of 30 closely related L. monocytogenes strains were first used for the phylogenetic analysis of L. monocytogenes EGD-e with mediocre-and low-quality reads. The Redbean assembly of low-quality reads was not included due to the error produced by CSI Phylogeny when this assembly with an inaccurate genome size of 30,776 bp was processed. As shown in Figure 1, the assemblies formed a single monophyletic clade with the reference genome and closely related strains, irrespective of the assemblies of mediocre-or low-quality reads, although the Flye assembly of mediocre-quality reads formed a dispersed paraphyletic clade relative to the major clade of the reference genome and other assemblies. The Miniasm/Racon and Raven assemblies of mediocre-quality reads had the smallest distance from the reference genome. Interestingly, the Miniasm/Racon assembly of low-quality reads with an inaccurate genome size of 542,915 bp was also on the clade where the reference genome was located. When 30 distantly related L. monocytogenes strains were also included in the phylogenetic analysis of L. monocytogenes EGD-e, a similar clade topology of the monophyletic clade that included the reference genome and assemblies were observed (Figure 2), where the Miniasm/Racon and Raven assemblies still had the smallest distance from the reference genome.

Whole-Genome Phylogeny
Taylor et al. [27] used 23 closely related Salmonella strains (number of SNPs < 500) to build a whole-genome phylogenetic tree with the Miniasm/Racon assembly of the Oxford Nanopore long reads of S. Bareilly CFSAN000189, where they observed a congruent topology between trees built with the SPAdes assembly of Illumina short reads and the Miniasm/Racon assembly. In our work, both closely and distantly related strains were included when evaluating the performance of Oxford Nanopore long-read assemblies in the phylogenetic analyses. A total of 30 closely related L. monocytogenes strains were first used for the phylogenetic analysis of L. monocytogenes EGD-e with mediocre-and low-quality reads. The Redbean assembly of low-quality reads was not included due to the error produced by CSI Phylogeny when this assembly with an inaccurate genome size of 30,776 bp was processed. As shown in Figure 1, the assemblies formed a single monophyletic clade with the reference genome and closely related strains, irrespective of the assemblies of mediocre-or lowquality reads, although the Flye assembly of mediocre-quality reads formed a dispersed paraphyletic clade relative to the major clade of the reference genome and other assemblies. The Miniasm/Racon and Raven assemblies of mediocre-quality reads had the smallest distance from the reference genome. Interestingly, the Miniasm/Racon assembly of low-quality reads with an inaccurate genome size of 542,915 bp was also on the clade where the reference genome was located. When 30 distantly related L. monocytogenes strains were also included in the phylogenetic analysis of L. monocytogenes EGD-e, a similar clade topology of the monophyletic clade that included the reference genome and assemblies were observed (Figure 2), where the Miniasm/Racon and Raven assemblies still had the smallest distance from the reference genome.  Whole-genome phylogenetic tree of Oxford Nanopore long-read assemblies of Listeria monocytogenes EGD-e with mediocre-and low-quality reads using different long-read assemblers, as aligned to 30 distantly related L. monocytogenes strains selected based on the single-nucleotide polymorphisms (SNPs) strategy (number of SNPs > 500) and compared with the reference genome.

Figure 2.
Whole-genome phylogenetic tree of Oxford Nanopore long-read assemblies of Listeria monocytogenes EGD-e with mediocre-and low-quality reads using different long-read assemblers, as aligned to 30 distantly related L. monocytogenes strains selected based on the single-nucleotide polymorphisms (SNPs) strategy (number of SNPs > 500) and 30 closely related L. monocytogenes strains selected based on the SNP (number of SNPs < 500) and core-genome multilocus sequence typing (cgMLST) (different alleles < 500) strategies and compared with the reference genome. Whole-genome phylogenetic tree of Oxford Nanopore long-read assemblies of Listeria monocytogenes EGD-e with mediocre-and low-quality reads using different long-read assemblers, as aligned to 30 distantly related L. monocytogenes strains selected based on the single-nucleotide polymorphisms (SNPs) strategy (number of SNPs > 500) and 30 closely related L. monocytogenes strains selected based on the SNP (number of SNPs < 500) and core-genome multilocus sequence typing (cgMLST) (different alleles < 500) strategies and compared with the reference genome.
S. Bareilly CFSAN000189 assemblies with real reads and its reference genome were aligned to 30 closely related S. Bareilly strains ( Figure 3). All 30 closely related strains and the reference genome formed a single monophyletic clade, whereas the assemblies formed a dispersed paraphyletic group sister to the reference genome clade. Similar results were obtained for the tree of C. jejuni NCTC 11168 with real reads built with 11 closely related C. jejuni strains selected based on the SNP strategy (Figure 4), where the reference genome formed a monophyletic group with the closely related strains and the assemblies were a non-monophyletic group forming multiple independent clades. When 20 distantly related C. jejuni strains were included in the phylogenetic analysis of C. jejuni NCTC 11168, the assemblies were still placed on different clades compared to the reference genome ( Figure 5). Although biased results were observed across all assemblies of both S. Bareilly CFSAN000189 and C. jejuni NCTC 11168, the Miniasm/Racon and Raven assemblies always had the smallest distance from the reference genomes. We then included 20 Campylobacter strains of other species in the phylogenetic analysis of C. jejuni NCTC 11168 ( Figure 6). Interestingly, all assemblies except the Redbean and Shasta assemblies were on the clade where the reference genome was located. The Miniasm/Racon and Raven assemblies still displayed the smallest distance from the reference genome. S. Bareilly CFSAN000189 assemblies with real reads and its reference genome were aligned to 30 closely related S. Bareilly strains ( Figure 3). All 30 closely related strains and the reference genome formed a single monophyletic clade, whereas the assemblies formed a dispersed paraphyletic group sister to the reference genome clade. Similar results were obtained for the tree of C. jejuni NCTC 11168 with real reads built with 11 closely related C. jejuni strains selected based on the SNP strategy ( Figure  4), where the reference genome formed a monophyletic group with the closely related strains and the assemblies were a non-monophyletic group forming multiple independent clades. When 20 distantly related C. jejuni strains were included in the phylogenetic analysis of C. jejuni NCTC 11168, the assemblies were still placed on different clades compared to the reference genome ( Figure 5). Although biased results were observed across all assemblies of both S. Bareilly CFSAN000189 and C. jejuni NCTC 11168, the Miniasm/Racon and Raven assemblies always had the smallest distance from the reference genomes. We then included 20 Campylobacter strains of other species in the phylogenetic analysis of C. jejuni NCTC 11168 ( Figure 6). Interestingly, all assemblies except the Redbean and Shasta assemblies were on the clade where the reference genome was located. The Miniasm/Racon and Raven assemblies still displayed the smallest distance from the reference genome. Figure 3. Whole-genome phylogenetic tree of Oxford Nanopore long-read assemblies of Salmonella Bareilly CFSAN000189 with real reads using different long-read assemblers, as aligned to 30 closely related S. Bareilly strains selected based on the single-nucleotide polymorphisms (SNPs) (number of SNPs < 500) and whole-genome multilocus sequence typing (wgMLST) strategies (different alleles < 500) and compared with the reference genome. Figure 3. Whole-genome phylogenetic tree of Oxford Nanopore long-read assemblies of Salmonella Bareilly CFSAN000189 with real reads using different long-read assemblers, as aligned to 30 closely related S. Bareilly strains selected based on the single-nucleotide polymorphisms (SNPs) (number of SNPs < 500) and whole-genome multilocus sequence typing (wgMLST) strategies (different alleles < 500) and compared with the reference genome. Figure 4. Whole-genome phylogenetic tree of Oxford Nanopore long-read assemblies of Campylobacter jejuni NCTC 11168 with real reads using different long-read assemblers, as aligned to 11 closely related C. jejuni strains selected based on the single-nucleotide polymorphisms (SNPs) strategy (number of SNPs < 500) and compared with the reference genome. Whole-genome phylogenetic tree of Oxford Nanopore long-read assemblies of Campylobacter jejuni NCTC 11168 with real reads using different long-read assemblers, as aligned to 11 closely related C. jejuni strains selected based on the single-nucleotide polymorphisms (SNPs) strategy (number of SNPs < 500) and compared with the reference genome. Int. J. Mol. Sci. 2020, 21, x FOR PEER REVIEW 20 of 32 Figure 5. Whole-genome phylogenetic tree of Oxford nanopore long-read assemblies of Campylobacter jejuni NCTC 11168 with real reads using different long-read assemblers, as aligned to 11 closely related (number of SNPs < 500) and 20 distantly related C. jejuni strains (number of SNPs > 500) selected based on the single-nucleotide polymorphisms (SNPs) strategy and compared with the reference genome.

Figure 5.
Whole-genome phylogenetic tree of Oxford nanopore long-read assemblies of Campylobacter jejuni NCTC 11168 with real reads using different long-read assemblers, as aligned to 11 closely related (number of SNPs < 500) and 20 distantly related C. jejuni strains (number of SNPs > 500) selected based on the single-nucleotide polymorphisms (SNPs) strategy and compared with the reference genome.
The higher error rates of Oxford Nanopore long reads may confound the phylogenetic differences between the assemblies and closely related strains selected based on the SNP and core-genome MLST (cgMLST) or whole-genome MLST (wgMLST) strategies, further confirming our observations that Oxford Nanopore long-read assemblies contained SNPs (Supplementary Tables S2, S4 and S6) and also produced inaccurate MLST results (Tables 9-11) relative to the reference genomes. We identified some recurrent phylogenetic patterns of long-read assemblies that could potentially be addressed in the future, as continued improvements in nanopore chemistry and basecalling would systematically mitigate the high numbers of errors to achieve a more accurate phylogenetic inference. Figure 6. Whole-genome phylogenetic tree of Oxford nanopore long-read assemblies of Campylobacter jejuni NCTC 11168 with real reads using different long-read assemblers, as aligned to 11 closely related (number of SNPs < 500), 20 distantly related C. jejuni strains (number of SNPs > 500) selected based on the single-nucleotide polymorphisms (SNPs) strategy, and 20 Campylobacter strains of other species and compared with the reference genome.
The higher error rates of Oxford Nanopore long reads may confound the phylogenetic differences between the assemblies and closely related strains selected based on the SNP and coregenome MLST (cgMLST) or whole-genome MLST (wgMLST) strategies, further confirming our observations that Oxford Nanopore long-read assemblies contained SNPs (Supplementary Tables S2,  S4, S6) and also produced inaccurate MLST results (Tables 9-11) relative to the reference genomes. We identified some recurrent phylogenetic patterns of long-read assemblies that could potentially be addressed in the future, as continued improvements in nanopore chemistry and basecalling would systematically mitigate the high numbers of errors to achieve a more accurate phylogenetic inference.

Pan Genomes
A total of 20 distantly related P. aeruginosa strains were included in the pan-genome analysis of P. aeruginosa PAO1 with mediocre-and low-quality reads. Raven was the most effective assembler Whole-genome phylogenetic tree of Oxford nanopore long-read assemblies of Campylobacter jejuni NCTC 11168 with real reads using different long-read assemblers, as aligned to 11 closely related (number of SNPs < 500), 20 distantly related C. jejuni strains (number of SNPs > 500) selected based on the single-nucleotide polymorphisms (SNPs) strategy, and 20 Campylobacter strains of other species and compared with the reference genome.

Pan Genomes
A total of 20 distantly related P. aeruginosa strains were included in the pan-genome analysis of P. aeruginosa PAO1 with mediocre-and low-quality reads. Raven was the most effective assembler for the pan-genome analysis of P. aeruginosa PAO1 with mediocre-quality reads, which displayed the numbers of core (core and soft-core) and accessory (shell and cloud) genes that were most similar to the reference genome (Figure 7). The pan genomes of the Raven assembly consisted of a total of 20,583 genes with 2615 core genes (12.7%) and 17,968 accessory genes (87.3%). Similarly, there were a total of 19,673 genes with 2866 core genes (14.6%) and 16,807 accessory genes (85.4%) in the pan genomes of the reference genome. Our results thus demonstrate that Raven could tolerate the inaccuracy of Oxford Nanopore long reads. Raven was closely followed by Miniasm/Racon that was also able to achieve similar pan-genome patterns with the reference genome. Canu, Flye, and Shasta were moderate performers for the pan-genome analysis, while Redbean was the least effective, producing inaccurate numbers of core genes (4.6%) and accessory genes (95.4%), respectively. of 19,673 genes with 2866 core genes (14.6%) and 16,807 accessory genes (85.4%) in the pan genomes of the reference genome. Our results thus demonstrate that Raven could tolerate the inaccuracy of Oxford Nanopore long reads. Raven was closely followed by Miniasm/Racon that was also able to achieve similar pan-genome patterns with the reference genome. Canu, Flye, and Shasta were moderate performers for the pan-genome analysis, while Redbean was the least effective, producing inaccurate numbers of core genes (4.6%) and accessory genes (95.4%), respectively. Figure 7. Pan genomes of Oxford Nanopore long-read assemblies of Pseudomonas aeruginosa PAO1 with mediocre-(a) and low-quality (b) reads using different long-read assemblers and 20 distantly related P. aeruginosa strains selected based on the single-nucleotide polymorphisms (SNPs) strategy (number of SNPs > 500) and compared with the reference genome. Figure 7. Pan genomes of Oxford Nanopore long-read assemblies of Pseudomonas aeruginosa PAO1 with mediocre-(a) and low-quality (b) reads using different long-read assemblers and 20 distantly related P. aeruginosa strains selected based on the single-nucleotide polymorphisms (SNPs) strategy (number of SNPs > 500) and compared with the reference genome.
Regarding the pan-genome analysis of the assemblies of low-quality reads, the Flye, Miniasm/Racon, and Redbean assemblies did not produce accurate numbers of core genes, although their total numbers of genes were similar to the reference genome, which could be attributed to their inaccurate genome sizes. For the pan genome of the Raven assembly of low-quality reads, we observed a dramatic increase in the total number of genes (28,119) compared to that of mediocre-quality reads, with a decrease in the number of core genes (1696) and an increase in the number of accessory genes (26,423).
Six closely related strains of E. coli O157:H7 CFSAN076619 with real reads were found based on the SNP and wgMLST strategies, which were thus included in its pan-genome analysis. The pan genome of the reference genome and the six strains consisted of 5790 genes with 4595 core genes (79.4%) and 1195 accessory genes (20.6%) (Figure 8). The Miniasm/Racon and Raven assemblies had numbers of core genes most similar to the reference genome compared to other assemblies, although the pan genomes of all assemblies contained large numbers of accessory genes. One key reason for this poor performance is the inherently limited accuracy of Oxford Nanopore long reads, as also revealed by our SNP and indel analyses, demonstrating that the degree of errors in an assembly can greatly affect conclusions about gene presence and absence beyond just the inability to resolve genomic structures. SNPs and indels detected in the assemblies of Oxford Nanopore long reads could introduce truncated genes that can produce a large number of misannotated genes, which increased the numbers of accessory genes in the pan genomes [18]. Our pan-genome analyses likely reflect the difficulty in using highly error-prone Oxford Nanopore long reads for accurately annotating genes due to gene truncation and misplaced start sites. Another aspect worth noting is that this limitation of Oxford Nanopore sequencing could become more remarkable when closely related strains were included in the pan-genome analysis. Our pan-genome analyses thus again highlight the limitation of Oxford Nanopore sequencing to produce accurate long reads. Overcoming this existing deficiency could greatly boost the annotations of many clinically and microbiologically important genomic regions in bacterial pathogens. Despite all these observations using real reads, Miniasm/Racon and Raven were comparable for the pan-genome analysis and performed the best among all assemblers tested, although both of their pan genomes still contained over 6000 accessory genes.
genome of the reference genome and the six strains consisted of 5790 genes with 4595 core genes (79.4%) and 1195 accessory genes (20.6%) (Figure 8). The Miniasm/Racon and Raven assemblies had numbers of core genes most similar to the reference genome compared to other assemblies, although the pan genomes of all assemblies contained large numbers of accessory genes. One key reason for this poor performance is the inherently limited accuracy of Oxford Nanopore long reads, as also revealed by our SNP and indel analyses, demonstrating that the degree of errors in an assembly can greatly affect conclusions about gene presence and absence beyond just the inability to resolve genomic structures. SNPs and indels detected in the assemblies of Oxford Nanopore long reads could introduce truncated genes that can produce a large number of misannotated genes, which increased the numbers of accessory genes in the pan genomes [18]. Our pan-genome analyses likely reflect the difficulty in using highly error-prone Oxford Nanopore long reads for accurately annotating genes due to gene truncation and misplaced start sites. Another aspect worth noting is that this limitation of Oxford Nanopore sequencing could become more remarkable when closely related strains were included in the pan-genome analysis. Our pan-genome analyses thus again highlight the limitation of Oxford Nanopore sequencing to produce accurate long reads. Overcoming this existing deficiency could greatly boost the annotations of many clinically and microbiologically important genomic regions in bacterial pathogens. Despite all these observations using real reads, Miniasm/Racon and Raven were comparable for the pan-genome analysis and performed the best among all assemblers tested, although both of their pan genomes still contained over 6000 accessory genes. Figure 8. Pan genomes of Oxford Nanopore long-read assemblies of Escherichia coli O157:H7 CFSAN076619 with real reads using different long-read assemblers and six closely related E. coli O157:H7 strains selected based on the whole-genome multilocus sequence typing (wgMLST) strategy (different alleles < 500) and compared with the reference genome. Figure 8. Pan genomes of Oxford Nanopore long-read assemblies of Escherichia coli O157:H7 CFSAN076619 with real reads using different long-read assemblers and six closely related E. coli O157:H7 strains selected based on the whole-genome multilocus sequence typing (wgMLST) strategy (different alleles < 500) and compared with the reference genome.

Raven and Miniasm/Racon
Based on the performance in various genomic analyses, overall, the most robust and accurate long-read assembler was Raven, closely followed by Miniasm/Racon. Interestingly, Raven is built upon modules of Miniasm and Racon. However, Raven executes novel algorithms for the overlap and layout phases of genome assembly that increase the contiguity of the final assembly [14]. Miniasm provides an ultrafast de novo assembly for long noisy reads, although it does not have a consensus step [12]. While the lack of error correction results in assemblies with a significant error rate, it can be enhanced with a suitably efficient consensus phase using an additional polishing step [28,29]. Racon contains a standalone consensus module for raw uncorrected assemblies of long reads generated by rapid assembly methods that do not include a consensus step [13]. Senol Cali et al. [30] conducted a review to analyze state-of-the-art tools using Oxford Nanopore long reads in terms of accuracy, speed, memory efficiency, and scalability. After a comprehensive analysis, they recommended that Miniasm and Racon should be used for assembly and polishing, respectively. Our findings further demonstrate the effectiveness of Miniasm/Racon-based assembly processes for Oxford Nanopore long reads, suggesting that assembly strategies following that algorithmic model may be most fruitful in the future. We acknowledge that not all currently available assemblers were included because the field of long-read assembly continues to evolve. The assemblers chosen for benchmarking have been widely used in the published literature of genome assembly beyond the initial release of the software, which also differ in their underlying assembly algorithms. As such the comparisons were more associated with their algorithms. We chose to benchmark the long-read assemblers with their default parameters and recommended settings. Future optimization of these parameters and settings before implementation could potentially improve their assembly algorithms. We can anticipate that with further development of Oxford Nanopore sequencing technology, read quality will increase, which could make assembly using Oxford Nanopore long reads more accurate and applicable to a wide range of bacterial pathogens.

Simulated Oxford Nanopore Long Reads
To assess whether long-read assemblers could tolerate problems encountered in real error-prone Oxford Nanopore long reads, the complete genomes of 10 species (10 strains) of bacterial pathogens, spanning a wide range of genome sizes and GC contents, were selected from the National Center for Biotechnology Information (NCBI) ( Table 10), which were used as the reference genomes in our study.
Badread 0.1.5 [31] was used based on the Nanopore error model to mimic simulated Oxford Nanopore long reads of mediocre quality, with a mean fragment size of 15,000 bp, fragment size standard deviation of 13,000 bp, mean identity of 85, max identity of 95, identity standard deviation of 5, and coverage of 50×. The chimera join rate, junk read rate, and random read rate of each simulated mediocre-quality dataset were adjusted to 1%. Oxford Nanopore long reads of low quality were also simulated using Badread by artificially introducing more chimeras, low-quality regions, and systematic basecalling errors, with a glitch rate of 1000, glitch size of 100, glitch skip of 100, mean identity of 75, max identity of 90, identity standard deviation of 8, and coverage of 50×. The chimera join rate, junk read rate, and random read rate of each simulated low-quality dataset were adjusted to 10%, 5%, and 5%, respectively. Oxford Nanopore ligation adapters were added to the start and end of each read using Badread, with a start adapter rate of 90 and start adapter amount of 60, and an end adapter rate of 50 and end adapter amount of 20. Start and end adapter sequences were AATGTACTTCGTTCAGTTACGTATTGCT and GCAATACGTAACTGAACGAAGT, respectively [31].

Real Oxford Nanopore Long Reads
Ten species (10 strains) of bacterial pathogens with publicly available complete genomes from the NCBI were tested for real Oxford Nanopore long reads, together covering a wide range of genome sizes and GC contents ( Table 11). The complete genomes were used as the reference genomes in this study. Oxford Nanopore long reads were downloaded from the Sequence Read Archive (SRA) of the NCBI.

Long-Read Assemblers
We assembled Oxford Nanopore long reads using six long-read assemblers with default parameters, namely Canu 1.8, Flye 2.3.7, Miniasm 0.2/Racon 1.3.1.1, Raven 0.0.8, Redbean 2.0, and Shasta 0.4.0. Miniasm/Racon was applied with two rounds of Racon polishing after the initial Miniasm assembly. Canu, Flye, and Redbean require a specified genome size as input. Flye, Miniasm/Racon, and Shasta indicate circularity by producing GFA files of their final assemblies, while Canu signals circularity via 'suggestCircular' in the header lines of FASTA files. However, Raven and Redbean do not indicate whether a contig is circular or not.

Computational Environments
Assemblies with Raven and Shasta were carried out on the Linux operating system of Ubuntu 18.04.4 LTS on a computer with 16 threads of CPU and 32 GB of RAM. Twelve threads of CPU were allocated to each assembler in the option of the number of threads used. Canu, Flye, Miniasm/Racon, and Redbean were available on the Amazon Web Services (AWS)-based GalaxyTrakr platform developed by the U.S. Food and Drug Administration (FDA) and intended for use by GenomeTrakr laboratories [32].

Assessment of Genome Completeness and Accuracy
Assembly quality was evaluated using Quast 5.0.2 [33] by computing several metrics, including the number of contigs, total length (bp), GC content, and indels. The number of indels was reported as the number of indels per one million bp of the reference genome. BUSCO 4.0.6 [34] was used to provide a quantitative assessment of genome completeness, with 0.01 as the E-value cutoff for BLAST searches and three candidate regions to consider. The results were expressed as complete BUSCOs that represent the fraction of the expected gene complement with full-length reading frames. CSI Phylogeny 1.4 [35] was used to call SNPs of each long-read assembly relative to the corresponding reference genome. Default settings were used, with 10× as the minimum depth at SNP positions, 10% as the minimum relative depth at SNP positions, 10 bp as the minimum distance between SNPs, 30 as the minimum SNP quality, 25 as the minimum read mapping quality, and 1.96 as the minimum Z-score. The reference genome of each strain was used for SNP calling. The number of SNPs was expressed as the number of SNPs per one million bp of the reference genome.

MLST
MLST was performed with mlst 2.19.0 (https://github.com/tseemann/mlst) that incorporates the components of the PubMLST database [40] through comparing assemblies against traditional PubMLST typing schemes based on seven housekeeping genes using default settings, with 95% as the minimum identity of full allele, 10% as the minimum coverage of partial allele, and 50 as the minimum score to match a scheme.

Whole-Genome Phylogenetic Analyses
Listeria monocytogenes EGD-e was used for the phylogenetic analysis of the assemblies of simulated reads. We first selected 30 closely related L. monocytogenes strains (Supplementary Table S13) based on the SNP (number of SNPs < 500) and cgMLST (different alleles < 500) strategies using BacWGSTdb 2.0 [41]. We then added 30 distantly related L. monocytogenes strains (Supplementary Table S14) selected based on the SNP strategy (number of SNPs > 500) using BacWGSTdb to the 30 closely related L. monocytogenes strains.
Salmonella Bareilly CFSAN000189 and Campylobacter jejuni NCTC 11168 were used for the phylogenetic analysis of the assemblies of real reads. To perform the phylogenetic analysis, long-read assemblies of these two strains were aligned to datasets of 30 closely related S. Bareilly (Supplementary  Table S15) and 11 closely related C. jejuni strains (Supplementary Table S16), respectively. The closely related datasets were selected based on the SNP (number of SNPs < 500) and wgMLST strategies (different alleles < 500) using BacWGSTdb. Twenty distantly related C. jejuni strains (number of SNPs > 500) and 20 Campylobacter strains of other species were also included in the phylogenetic analysis of C. jejuni NCTC 11168 (Supplementary Table S17).
The whole-genome phylogenetic analyses were performed with all long-read assemblies and their corresponding reference genomes, as aligned to the selected datasets. CSI Phylogeny was used with default settings as previously described to call SNPs of long-read assemblies and to calculate phylogenetic relationships based on the concatenated alignment of the high-quality SNPs. L. monocytogenes AT3E (RefSeq assembly accession: GCF_002557735.1), S. typhimurium LT2 (RefSeq assembly accession: GCF_000006945.2), and C. jejuni isolate_W1 (RefSeq assembly accession: GCF_002179165.1) served as the reference genomes for L. monocytogenes EGD-e, S. Bareilly CFSAN000189, and C. jejuni NCTC 11168, respectively. The inferred maximum-likelihood whole-genome phylogeny was visualized as a cladogram with Geneious Prime 2020.1.2. (Biomatters, Ltd., Auckland, New Zealand).
Pseudomonas aeruginosa PAO1 and Escherichia coli O157:H7 CFSAN076619 were used for the pan-genome analyses of simulated and real reads, respectively. Thirty distantly related P. aeruginosa PAO1 (Supplementary Table S18) were used for the pan-genome analysis of P. aeruginosa PAO1. For the pan-genome analysis of E. coli O157:H7 CFSAN076619, six closely related E. coli O157:H7 strains (Supplementary Table S19) were selected based on the SNP and wgMLST strategies using BacWGSTdb.

Statistical Analyses
The Wilcoxon signed-rank test was performed using SigmaPlot 14.0 (Systat Software Inc., San Jose, CA, USA) to determine whether significant differences (p < 0.05) existed between the reference genomes and completed long-read assemblies in genome size, GC content, complete BUSCOs, and the number of virulence genes, among completed long-read assemblies in numbers of SNPs and indels. This test was also conducted to determine whether there were significant differences (p < 0.05) between the assemblies of mediocre-and low-quality reads in genome size, GC content, complete BUSCOs, and the number of virulence genes, SNPs, and indels.

Conclusions
The significance of long-read assembly will continue to grow as Oxford Nanopore sequencing is becoming more and more widely used in the genomics of bacterial pathogens. An ideal long-read assembler for bacterial pathogens should efficiently complete the assembly process and remain robust against a wide range of systematic error-prone long reads. Most importantly, the genome assembly generated should be able to provide accurate genomic information for downstream pathogen identification, genotypic prediction, genome-wide annotation, and phylogenetic inference.
Our benchmarking using both simulated (mediocre-and low-quality) and real reads highlights the salient fact that Oxford Nanopore sequencing errors may cause difficulties in resolving the phylogenetic differences in some cases. Similarly, our pan-genome analyses also demonstrate the limitation of using error-prone Oxford Nanopore long reads to produce a high-quality assembly for accurate genome annotation. These problems could be addressed with the improvement of nanopore chemistry and basecalling. Overall, the most robust and accurate long-read assembler was Raven, closely followed by Miniasm/Racon, as revealed by its performance in various genomic analyses, although Raven does not indicate whether a contig is circular or not. With the improvement of sequencing technologies and assembly algorithms, defining an optimal long-read assembly approach for genomic analyses of bacterial pathogens is a continuous process.
To decrease the assembly errors caused by a single assembler to the desired level, it is warranted to rely on multiple assemblers to perform standalone assemblies and compare their independent results in genomic analyses. This could provide the opportunity for additional data validation, as confidence in one assembly becomes greater only when its results are consistent with those of other assemblies.