Evaluation of WHO catalog of mutations and five WGS analysis tools for drug resistance prediction of Mycobacterium tuberculosis isolates from China

ABSTRACT The continuous advancement of molecular diagnostic techniques, particularly whole-genome sequencing (WGS), has greatly facilitated the early diagnosis of drug-resistant tuberculosis patients. Nonetheless, the interpretation of results from various types of mutations in drug-resistant-associated genes has become the primary challenge in the field of molecular drug-resistance diagnostics. In this study, our primary objective is to evaluate the diagnosis accuracy of the World Health Organization (WHO) catalog of mutations and five WGS analysis tools (PhyResSE, Mykrobe, TB Profiler, Gen-TB, and SAM-TB) in drug resistance to 10 anti-Mycobacterium tuberculosis (MTB) drugs. We utilized the data of WGS collected between 2014 and 2017 in Zhejiang Province, consisting of 110 MTB isolates as detailed in our previous study. Based on phenotypic drug susceptibility testing (DST) results using the proportion method on Löwenstein-Jensen medium with antibiotics, we evaluated the predictive accuracy of genotypic DST obtained by these tools. The results revealed that the WHO catalog of mutations and five WGS analysis tools exhibit robust predictive capabilities concerning resistance to isoniazid, rifampicin, ethambutol, streptomycin, amikacin, kanamycin, and capreomycin. Notably, Mykrobe, SAM-TB, and TB Profiler demonstrate the most accurate predictions for resistance to pyrazinamide, prothionamide, and para-aminosalicylic acid, respectively. These findings are poised to significantly guide and influence future clinical treatment strategies and resistance monitoring protocols. IMPORTANCE Whole-genome sequencing (WGS) has the potential for the early diagnosis of drug-resistant tuberculosis. However, the interpretation of mutations of drug-resistant-associated genes represents a significant challenge as the amount and complexity of WGS data. We evaluated the accuracy of the World Health Organization catalog of mutations and five WGS analysis tools in predicting drug resistance to first-line and second-line anti-TB drugs. Our results offer clinicians guidance on selecting appropriate WGS analysis tools for predicting resistance to specific anti-TB drugs.

accurate, has limitations such as complexity, time consumption, and technical demands, which restrict its widespread use (2).In recent years, rapid molecular assays using molecular biological techniques have emerged as promising alternatives.The Xpert MTB/RIF or Xpert Ultra is a PCR diagnostic test that detects rpoB gene mutation in MTB related to rifampicin (RFP) resistance, offering higher sensitivity and shorter diagnostic time compared to DST (3).Nevertheless, this method can only identify a limited number of resistance-associated gene mutations, and its results may not be comprehensive or entirely reliable.More recently, genotypic resistance prediction from MTB sequences using whole-genome sequencing (WGS) has been in rapid development.WGS not only enables faster determination of drug resistance compared to conventional DST and molecular assays but also accurately predicts resistance to various anti-TB drugs without requiring specialized infrastructure (4).While sequencing technology has matured, data analysis remains a challenge for predicting drug resistance using WGS technol ogy.Fortunately, several tools have been developed to aid in the accurate and rapid identification and prediction of DR-TB, including PhyResSE (5), Mykrobe (6), TB Profiler (7), Gen-TB (8), and SAM-TB (9).These tools vary in their scope and specificity, from generic to more refined applications.Besides detecting drug-resistance mutations and identifying MTB lineages, they can be also used for species identification of nontubercu lous mycobacteria (NTM) or combined these two functions (9).Moreover, the WHO has published a catalog of MTB complex mutations associated with drug resistance, serving as a global standard for interpreting molecular information on resistance predictions (10).The performance of the WHO mutations catalog or WGS online tools to predict drug resistance depends on the integrity and accuracy of the drug-resistant mutation database employed.However, it is worth noting that variations in DR-TB mutations may exist across different countries, regions, ethnic groups, and populations.Further studies are needed to validate the predictive value of WGS analysis tools for different populations and regions.
In this study, our main objective was to evaluate the accuracy of the WHO muta tions catalog and five WGS analysis tools (PhyResSE, Mykrobe, TB Profiler, Gen-TB, and SAM-TB) in predicting resistance to both first-line and second-line drugs.To achieve this, we utilized DST and WGS data from a total of 110 MTB isolates, which had been previously published in our earlier study (11).The aim was to compare the predictions made by these tools against the known resistance profiles obtained through DST and WGS analysis of the isolates.This assessment was critical in determining the reliability and effectiveness of the WHO catalog and the WGS tools in accurately predicting drug resistance in MTB, thereby contributing to the advancement of TB treatment and management strategies.

Prediction of resistance to first-line drugs
We assessed the predictive performance of the WHO catalog of mutations (first and second versions) and five WGS online analysis tools by phenotypic culture-based DST results (Table 1; Fig. 1 and 2).WHO catalog of mutations and five online analysis tools all showed high performance in predicting the specificity of isoniazid (INH) and RFP resistance, with all achieving 100% accuracy.However, there were some differences in sensitivity.Among them, TB Profiler exhibited the highest sensitivity, while Gen-TB had the lowest sensitivity.Gen-TB showed the best specificity in predicting ethambutol (EMB) resistance, while TB Profiler exhibited the best sensitivity in predicting EMB resistance.All tools achieved a specificity of over 97% in predicting streptomycin (SM) resistance and a sensitivity of over 86% (except for PhyResSE, which had a sensitivity of 76.81%).TB Profiler had the best sensitivity in predicting pyrazinamide (PZA) resistance, while the sensitivity predicted by other tools was 66% or lower.All tools predicted a specificity of over 91% for PZA resistance.Overall, the accuracy of tools in PZA resistance prediction varied significantly, with TB Profiler demonstrating the most accurate to others.

DISCUSSION
The early diagnosis of drug-resistant tuberculosis based on molecular diagnostic techniques is of crucial importance for the effective control of TB.In recent years, due to the rapid advancement of sequencing technologies, WGS has gained prominence as a vital method for drug-resistance testing in tuberculosis (12,13).Its key advantage lies in its ability to swiftly detect all known drug-resistant mutations.In 2018, the WHO recommended the use of WGS for the rapid diagnosis of drug-resistant tuberculosis (14).However, the main bottleneck of the current application of WGS lies in the analysis and interpretation of the massive data generated, as well as the lack of standardized, comprehensive coverage of gene mutations and their associations with drug resistance (15).Given this, numerous analysis tools have been developed to analyze WGS data and predict drug resistance.In 2021, the WHO also published the catalog of drug-resistant mutations to promote the mutual recognition of genotypic DST, phenotypic DST, and sequencing data, as well as to enhance the understanding of mutations associated with drug-resistant phenotypes.Moreover, WHO has recently published an update to the mutations catalog in 2023.In this study, we performed a comparison of the WHO catalog of mutations (version 1.0 and 2.0) and five user-friendly WGS analysis tools for the effectiveness of predicting drug resistance in MTB.The comparison involved 10 types of anti-tuberculosis drugs, including all first-line drugs (INH, RFP, EMB, SM, and PZA) and some second-line drugs (AM, KM, CM, PTO, and PAS).Overall, WGS tools demonstrate better prediction performance for first-line drugs than second-line drugs.
Compared to the WHO catalog of mutations and other WGS analysis tools, Gen-TB has poor performance in predicting the sensitivity of INH and RFP resistance.The catalog of INH-resistant mutations Gen-TB used only includes katG_S315N, katG_S315T, and fabG1_C-15T.In addition, resistant mutations of inhA gene (inhA_G154A, inhA_T770C, and inhA_C777T) are listed into the WHO catalog of mutations, which might explain why the WHO catalog of mutations achieves better sensitivity (83.17%) than Gen-TB (81.19%).Nearly all (95% or more) RFP-resistant strains harbor mutations within the 81 bp RFP-resistance-determining region in rpoB gene (16).The reason for the poor sensitivity of Gen-TB for RFB resistance is the lack of the mutations S431G, Q432L, D435Y, H445D, H445Q, and L452P in ropB gene which have been reported in previous studies (17)(18)(19).The different resistant mutations that were collected into the catalogs used for the interpretation of variants made the difference performance of WGS analysis tools.Specifically, we found that the predictive accuracy of WGS analysis tools for EMB, SM, and PZA showed large differences, for which the genetic basis of resistance is more complex than for RFP and INH.For example, PZA resistance is not only primarily associated with a mutation in pncA and its promoter region but also associated with a mutation in rpsA, panD, clpC1, and other unidentified mechanism of genes (20).
The WHO catalog of mutations and five WGS analysis tools achieve high accuracy in predicting resistance to AM, KM, and CM (consistency of more over 85%), but the results are not ideal for PTO.Only TB Profiler and SAM-TB could predict resistance for PAS with a good concordance rate of the phenotypic DST and WGS.PhyResSE cannot predict resistance for PTO, while the WHO catalog of mutations and other four tools perform good specificity but poor sensitivity for PTO resistance.
WHO has recently published an update to the mutations catalog (second version) in 2023 (21).We compared the first and second versions to investigate whether the updated catalog has improved the predictive ability for drug resistance.Following the update of the WHO catalog, we observed improvements in the prediction accuracy for first-line drugs INH, RFP, and PZA.This was evidenced by the inclusion of resist ance mutation sites such as inhA_G779T for INH, rpoB_431S_432QinsR for RFP, and pncA_C14W for PZA.The mutation inhA_G779T for INH, previously not considered a resistance mutation in the first version of the WHO catalog, is now recognized as such in the second version, possibly due to updates in WGS data analysis methodol ogy.The second version of the WHO catalog continues to retain variations with an allele frequency of ≥75% (which was 90% in the first version) for further association analysis, and it also provides additional assessment for variations with an allele frequency threshold reduced to 25% (22).Conversely, there were minimal alterations in the prediction results for second-line drugs.
Fluoroquinolones (FQs) are recognized for their critical role, especially in combination therapies with emerging drugs such as bedaquiline (BDQ) and linezolid (LZD) (23).The exclusion of FQs was a deliberate decision based on the retrospective nature of our data collection, which limited our access to certain drug classes.However, we acknowledge that this exclusion may introduce a limitation in our study's ability to fully elucidate the drug-resistance profiles predicted by WGS.The impact of not accounting for FQs could potentially affect the generalizability of our findings, particularly in settings where FQs are more commonly prescribed.Future research endeavors should aim to incorporate a broader spectrum of antimicrobial agents, including FQs, and refine WGS prediction models to better account for the intricate resistance profiles of Mtb strains (24).Additionally, our research also lacks the predictive capability of WGS for new anti-TB drugs (such as LZD and BDQ) resistance.This is partly because the acquisition time of our strains predates the launch of these new drugs in China, and no resistant strains were identified in the preliminary phenotypic results.Moreover, the mechanisms of resistance to these new drugs are not well understood, and there is poor consistency between existing genotype-based resistance analyses and phenotypic resistance (25).Therefore, this study did not discuss research on genotype-based prediction of resistance to these new drugs.
The present study provides a better understanding of the performance of the WHO catalog of mutations and five WGS analysis tools in predicting drug resistance in MTB.WHO catalog of mutations and five WGS analysis tools exhibit robust predictive capabilities concerning resistance to INH, RFP, EMB, SM, AM, KM, and CM.Mykrobe, SAM-TB, and TB Profiler demonstrate the most accurate predictions for resistance to PZA, PTO, and PAS, respectively.These findings will serve as critical points of reference and guidance for future clinical treatment and resistance monitoring.

Evaluation data set
Our data set consisted of 100 multi-drug resistant (MDR) strains and 10 non-MDR strains.These strains were randomly selected from a pool of over 6,000 clinical samples, among which 329 strains were identified as MDR.All strains were obtained from Wenzhou Sixth People's Hospital, Wenzhou Central Hospital Medical Group, Zhejiang Province, China, between 1 January 2014 and 30 June 2017.Both the phenotypic DST results and WGS data were available for these strains.This set of strains has been described in our previous study (11).Initially, a colloidal gold assay (Genesis Biodetection and Biocontrol Ltd., Hangzhou, Zhejiang Province, China) was routinely employed for the detection of the MPB64 antigen, allowing differentiation between MTB and NTM.All NTM isolates were systematically excluded from the study.Following this initial screening, DNA extraction was performed, and the presence of MTB was further validated through PCR amplification and Sanger sequencing of the 16s rRNA.Importantly, in cases where a patient presented with multiple isolates, caution was exercised by excluding the earlier isolates from the analysis.

Phenotypic DST and critical concentration
The DST of all clinical isolates and the reference strain MTB H37Rv to 13 anti-TB drugs were carried out according to the Clinical and Laboratory Standards Institute (CLSI) and WHO guidelines.All antibiotics, except PZA and PTO, were tested using the proportion method on a commercial Löwenstein-Jensen medium with antibiotics (Baso, Zhuhai, Guangzhou Province, China).The critical concentrations were 0.2 mg/L for INH, 40.0 mg/L for RFP, 2.0 mg/L for EMB, 4.0 mg/L for SM, 30.0 mg/L for AM, 30.0 mg/L for KM, 40.0 mg/L for CM, and 1.0 mg/L for PAS, respectively, according to the CLSI guidelines and WHO guidelines (26,27).The results were determined after 3 weeks of incubation at 37°C.The susceptibility of MTB to PZA and PTO was evaluated using an automated Mycobacterial Growth Indicator Tube 960system (Becton Dickinson Diagnostic Systems, Franklin Lakes, NJ, USA) according to the manufacturer's instructions at critical concentrations of 100.0 and 2.5 mg/L, respectively.The limitation of technical feasibility and reproducibility of the phenotypic DST of PZA, EMB, and PTO required the DST to be performed at least twice for these three drugs.If the two results were inconsistent, a third test was performed.All experiments using live MTB were performed in a biosafety level 2 plus laboratory.Among these strains, 100 of them were MDR strains, while the remaining strains were non-MDR strains.

TABLE 1
Performance of the WHO catalogs of mutations and five WGS analysis tools for predicting drug resistance of MTB (n = 110)

TABLE 1
Performance of the WHO catalogs of mutations and five WGS analysis tools for predicting drug resistance of MTB (n = 110) a

TABLE 1
Performance of the WHO catalogs of mutations and five WGS analysis tools for predicting drug resistance of MTB (n = 110) a

TABLE 1
Performance of the WHO catalogs of mutations and five WGS analysis tools for predicting drug resistance of MTB (n = 110) a