Inter-laboratory proficiency testing scheme for tumour next-generation sequencing in Ontario: a pilot study.

Background
A pilot inter-laboratory proficiency scheme for 5 Ontario clinical laboratories testing tumour samples for the Ontario-wide Cancer Targeted Nucleic Acid Evaluation (octane) study was undertaken to assess proficiency in the identification and reporting of next-generation sequencing (ngs) test results in solid tumour testing from archival formalin-fixed, paraffin-embedded (ffpe) tissue.


Methods
One laboratory served as the reference centre and provided samples to 4 participating laboratories. An analyte-based approach was applied: each participating laboratory received 10 ffpe tissue specimens profiled at the reference centre, with tumour site and histology provided. Laboratories performed testing per their standard ngs tumour test protocols. Items returned for assessment included genes and variants that would be typically reported in routine clinical testing and variant call format (vcf) files to allow for assessment of ngs technical quality.


Results
Two main aspects were assessed:■ Technical quality and accuracy of identification of exonic variants■ Site-specific reporting practicesTechnical assessment included evaluation of exonic variant identification, quality assessment of the vcf files to evaluate base calling, variant allele frequency, and depth of coverage for all exonic variants. Concordance at 100% was observed from all sites in the technical identification of 98 exonic variants across the 10 cases. Variability between laboratories in the choice of variants considered clinically reportable was significant. Of the 38 variants reported as clinically relevant by at least 1 site, only 3 variants were concordantly reported by all participating centres as clinically relevant.


Conclusions
Although excellent technical concordance for ngs tumour profiling was observed across participating institutions, differences in the reporting of clinically relevant variants were observed, highlighting reporting as a gap where consensus on the part of Ontario laboratories is needed.


INTRODUCTION
Next-generation sequencing (ngs) for molecular profiling of solid tumours is rapidly becoming standard-of-care testing in molecular laboratories in Canada because of the simultaneous yield of clinically useful genetic information, benefit of tissue preservation by avoiding sequential testing, and declining cost of ngs equipment and operations.
Proficiency testing (pt) schemes (also known as external quality assessment if offered by an external body), in which laboratories are assessed on blinded analysis of samples with known results, is an integral part of ensuring quality of ngs and other molecular tests 1,2 . However, a national pt scheme for molecular pathology laboratory testing (using ngs or any method) does not exist in Canada, likely because of the effort of establishing a scheme potentially subscribed to by only a small number of Canadian laboratories, and because of the effort of sample acquisition and pt assessment. In Ontario, all licensed and accredited clinical laboratories offering testing for patient care are required to participate in pt as mandated by the provincial accreditation body (Institute for Quality Management in Healthcare), based on the International Organization for Standardization 15189 standard. The Institute for Quality Management in Healthcare requires that, to maintain accreditation, clinical laboratories complete pt for 4 samples within a 12-month period for each clinical test. In the absence of a pt program offered by a Canadian organization for tumour molecular profiling, accredited laboratories must identify suitable alternatives, such as international pt programs or informal sample exchange, to meet the Ontario requirements.
A key consideration in the design of a pt scheme for tumour molecular profiling is the selection of sample source material. To perform both pre-analytic and analytic comparisons of laboratory proficiency, the optimal material is formalin-fixed, paraffin-embedded (ffpe) tumour tissue, because that sample source allows for an assessment of pre-analytic variables. However, obtaining ffpe tumour tissue for pt is often hampered by a small tumour amount and suboptimal quality of the available clinically relevant material. Tumour heterogeneity can also lead to potential differences in results. Although tracking of ffpe sections sent to participating laboratories is possible, that approach does not ameliorate the risk of error. An alternative approach is to use dna extracted from ffpe tissue 3 , with the inherent limitation that use of dna prevents identification of any potential issues related to the pre-analytic phase. Other source material could also be used, such as cell lines embedded in paraffin or synthetic dna controls, each with its own limitations. Any of those sample issues might lead to inappropriate discrepancies in pt testing results originating solely in the material sent within the pt scheme.
The other significant aspect in pt schemes is whether the scheme assesses only the technical aspects (for example, by requesting return of variants only) or also assesses the post-analytic clinical interpretation and reporting aspects (for example, by requesting that variant interpretations or mock clinical reports be returned). For testing of solid tumours, general guidelines about reporting aspects for laboratory tests are available 4 ; however, those guidelines might not be sufficiently detailed for pt schemes, which are often specific to a gene or a disease indication. Proficiency testing schemes might also request return of various types of data-for example, only the Human Genome Variation Society nomenclature for identified variants, or variants plus data files for data quality analysis, which typically compares data across laboratories rather than scoring based on an evaluative scheme.
In the present study, a pilot pt scheme (Figure 1) was implemented for Ontario laboratories participating in the Ontario-Wide Cancer Targeted Nucleic Acid Evaluation (octane) study, which is an ongoing prospective trial open at 5 academic cancer centres. The trial aims to enable genotype-drug matching through somatic ngs testing of ffpe solid tumour tissue from patients with advanced cancer and to facilitate clinical and genomic data-sharing. The pt scheme was designed as a pre-to post-analytic scheme, with dissemination of ffpe tumour material and return of variants, variant call format (vcf) files, and information about clinically reportable genes and variants. We highlight the successes and challenges of that approach to pt schemes for solid tumour molecular profiling in the Ontario context.

Participating Laboratories
Participating laboratories included the Princess Margaret Cancer Centre (pmcc), University Health Network, Toronto (which acted as the reference site for source materials and result evaluations); the Juravinski Cancer Centre (jcc), Hamilton; the London Health Sciences Centre (lhsc), London; The Ottawa Hospital (toh), Ottawa; and the Kingston Health Sciences Centre (khsc), Kingston. A laboratory director from each site was involved in the study design.

Tumour Tissue Specimens
From a cohort of ffpe tissues banked at the reference laboratory, 10 ffpe specimens of tumour tissue were chosen from resections in patients enrolled in the octane trial (NCT02906943 at https://ClinicalTrials.gov/) with approval from the Ontario Cancer Research Ethics Board (ID: 16-018). All 10 specimens had previously been tested by ngs at the reference laboratory. Specimens were chosen so as to provide to each of the 4 participating sites a variety of genes and variants, and to meet these additional criteria: tumour cellularity greater than or equal to 20% in the tumour area, and sufficient ffpe material to provide 2 unstained sections at 7 μm thickness and 1 slide stained with hematoxylin and eosin. Sections were numbered as cut, and the section numbers distributed to each laboratory were recorded to enable tracking of the location within the tumour block in case of tumour heterogeneity for variants. Unstained and uncircled sections were provided on air-dried, uncoated slides. Tumour site and histology as determined by a study pathologist at the reference site were also provided to the participating laboratories.

Molecular Profiling Assays
The participating laboratories used 2 ngs panels in testing dna samples. Ion Torrent PGM platform (Thermo Fisher) at each site. Those 4 laboratories were accredited by the Ontario provincial laboratory accreditation body (the Institute for Quality Management in Healthcare), and the remaining laboratory (pmcc) was accredited by the College of American Pathologists (cap) and was certified as meeting the U.S. Clinical Laboratory Improvement Amendments. Laboratories were instructed to not communicate results with other laboratories and to treat the samples in a manner similar to other clinical samples as much as possible for the entire workflow.

Variant Assessment and Reporting
For each of the 10 specimens, participating laboratories were requested to return the following information to the reference laboratory within 4 weeks (supplemental Appendix 1): ■ A list of genes for each tumour site the laboratory would consider to be clinically reportable from the panel in use ■ The variants identified in each of the 10 specimens that the laboratory considered to be clinically reportable, with variant interpretations ■ A list of variants that met laboratory-defined minimum technical quality metrics for high-quality variants (that is, all high-quality variants whether considered clinically reportable or not) ■ The vcf files from the relevant ngs panel test at each site The khsc provided vcf files that were pre-filtered according to their current laboratory practice; the jcc, toh, and lhsc provided unfiltered vcf files. Clinically reportable variants were defined as variants that would routinely be reported based on the clinical reporting practices for the octane study at that laboratory. Variant interpretations were requested to be performed and returned according to each laboratory's typical process for the octane study. Laboratories were requested to include Human Genome Variation Society nomenclature and any other nomenclature system typically used for reports generated for the octane study.

Assessment of Technical and Reporting Performance
A comparative analysis was conducted to assess concordance in the variant information returned from each participating site with the variants identified by the reference site. Because the reference site used different ngs chemistry, analyses included assessment of concordance between the participating sites, but discordance with the reference laboratory results. That approach was instituted to rule out the bias of considering the reference laboratory results to be "true." Variants were assessed by the accuracy with which reported variants were identified, including correct Human Genome Variation Society nomenclature. Technical quality metrics were assessed using the vcf files and BEDTools (version 2.23.0) 5 , with an intersection browser extensible data file created to identify overlapping regions in the UHN Hi5 panel and the Ion AmpliSeq Cancer Hotspot Panel v2, and to identify common and unique variants in the reference laboratory dataset and in each of the datasets from the 4 participating laboratories.
To assess reporting, the genes and variants that each laboratory provided as being clinically reportable were manually compared.

Reportable Genes by Tumour Site
Participating laboratories provided the list of genes typically reported clinically at their institution for each tumour site in the pt specimens, per their usual practice within the octane study. The list of reportable genes for each tumour site differed significantly between the participating centres ( Table i). Of the participating laboratories, 2 chose to report on all genes on their panel, and 1 elected to report only variants in genes that were routinely reported in clinical practice at their institution, regardless of tumour type.

Comparison of Base Calling and Quality Assessment of Raw Data
An analysis of the agreement in technical detection of variants identified in the vcf files for each case demonstrated 100% concordance (98 of 98 variants) in the identification of exonic variants from all sites for the 10 cases (Table ii). Of the 98 exonic variants detected, a subset was identified below the lower variant allele frequency (vaf) and quality thresholds defined for pt evaluation-that is, less than 5% vaf and less than 100× (pmcc) or 500× (jcc, lhsc, toh, khsc) coverage. Of those low-vaf or low-coverage variants, none was considered clinically reportable by more than 1 laboratory.

Variants Considered Clinically Reportable
The pt results provided by each institution included a list of variants in each case that were considered clinically reportable per routine practice in the octane study (Table iii). A high degree of variability in the variants considered clinically reportable was also observed, with only 3 variants from the 10 cases being concordantly reported by all 5 participating centres. Concordant reporting of 5 variants by 4 or more centres and of 10 variants by 3 or more centres was observed.

Interpretation of Clinically Reportable Variants
Variants considered clinically reportable were classified by 4 of the laboratories using a published somatic variant classification scheme. The joint guideline from the Association for Molecular Pathology (amp), the American Society of Clinical Oncology (asco), and cap (amp/asco/ cap) published by Li et al. 6 was applied by 3 laboratories, and the Sukhai et al. 7 guideline was used by 1 laboratory. Significant variability was observed in the classifications provided for specific variants (Table iii). For example, one laboratory indicated that the TP53 p.Leu252del variant identified in case 8 was a tier ii variant, while another indicated that the same variant was a tier iii variant. That same variant was classified as class 3A by a 3rd site and was not clinically reported by the remaining 2 laboratories. Similarly, the TP53 p.Ser127Phe variant identified in case 1 was reported as tier ii by 1 site and as tier iii by 1, with the other 3 sites not reporting it. Furthermore, 1 site chose to include variants classified as tier iii or iv according to the amp/asco/cap guideline as clinically reportable.

DISCUSSION
The present study set out to evaluate the performance of solid tumour molecular profiling at the 5 Ontario sites (jcc, lhsc, toh, khsc, and pmcc) that provide ngs molecular profiling for the octane study. An analyte-based pt approach was used, with ffpe tumour tissue being sent out, and information related to variants considered clinically relevant being returned, in an end-to-end evaluation of laboratory performance.
Although the use of ffpe tissue allows for an evaluation of the pre-analytic phase, it can also adversely affect other aspects of the pt scheme. At the reference laboratory, it was difficult to source sufficient ffpe tumour tissue material meeting all parameters specified in the Methods section for distribution to the 4 participant sites. Of the 10 samples,   X X X X X 4 were ovarian tumours, a site from which large surgical specimens with sufficient cellularity and lack of significant necrosis were more readily available. Only 2 lung cancer samples and 1 colorectal cancer sample were included because of the typically small tumour specimens obtained for those tumour types, although lung and colorectal cancers represent the tumours that most commonly undergo ngs testing as the standard of care in Ontario. That bias in sample selection is likely to have affected the interpretation or reporting aspects of this pt, because all participating laboratories routinely test lung and colorectal cancers, but only 3 of the 5 participating laboratories typically report on ovarian cancer. Another issue related to the use of ffpe tumour tissue is potential tumour heterogeneity, which can contribute to variability in the detection of variants. A recent survey involving 111 laboratories assessed inter-laboratory technical performance for ngs-based solid tumour oncology assays and identified substantial agreement (>98%) in the accuracy of detection for single nucleotide variants occurring at a vaf more than 15% 2 . Indeed, although we observed no difference in the final exonic variants identified in the present study (98 of 98), variability in the vaf and depth of coverage was evident during quality assessment of the vcf file data. Although that variability might be attributable to tumoural heterogeneity, it might also result from differences in pre-analytic sample processing or ngs quality or in differences in the sequencing technology, and further delineating the causes of those differences in vaf and coverage depth is not possible. It is also noteworthy that, although complete concordance was observed in the technical identification of the 98 variants, 28 of 98 variants were identified by 1 or more sites below the lower vaf and quality threshold cut-offs defined in the study.
With respect to the interpretation and reporting of variants in tumour molecular profiling, discrepancies were observed: only 3 variants were selected as clinically reportable by all participating sites, and only 10 variants were concordantly reported by 3 or more sites. In part, those results reflected site-specific interpretation of the instructions for the pt scheme (supplemental Table 1), because some laboratories reported only variants in genes routinely reported in clinical practice at that institution (rather than those reported in the octane study), regardless of tumour type. This site-specific reporting practice highlights the significant gap in the classification of what is considered a "clinically reportable" variant in the Ontario context and likely reflects practice in other Canadian provinces, because national standards for somatic variant interpretation do not currently exist. With respect to using published classification schemes to classify variants as "actionable," there was also no consensus concerning the classification scheme applied, with 3 sites applying the amp/asco/cap guideline 6 , 1 using the Sukhai et al. guideline 7 , and 1 not using a guideline. Of the sites that used the amp/asco/cap guideline, 2 included only tier i or ii variants as "clinically reportable"; another laboratory included tier i-iv variants. That observation underscores issues related to the understanding of the pt instructions, because tier iii and iv variants are generally not considered clinically actionable or reportable.  Variant was identified at below laboratory-defined thresholds.