CDx, NGS and regulation: five perspectives from the Pistoia Alliance

Companion diagnostics (CDx) are essential to the practice of precision medicine. Next-generation sequencing is an increasingly important tool in the development of CDx. However, for CDx to be deployed, many different biopharma industry sectors need to collaborate. This paper outlines some of the challenges and opportunities perceived by the biopharmaceutical industry, the Europe Molecular Quality Network, a regulatory agency, a notified body and a CDx service provider. Drug Discovery Today Volume 00, Number 00 July 2019 PERSPECTIVE


Introduction
The development of sequencing technology has accelerated enormously. Sequencing costs are below US$1000 per genome. It is now possible for just one sequencing machine to process up to 200 human genomes a day. The Wellcome Sanger Institute alone is capable of sequencing >2 Petabases (10 15 ) of data (or~20 000 human genome equivalents) per year, an 11x increase of throughput in the past 4 years. The availability of cheaper next-generation sequencing (NGS) has led to a rapid expansion in the number of large genome projects using hundreds of thousands, even millions, of genomes [1]. Several market reports and scientific reviews have shown the rapid growth in the use of biomarkers and companion diagnostics (CDx) in drug discovery and development. A recent report from Frost & Sullivan [2] suggested these tools would be one of the key growth areas in the developing precision medicine market and could be worth US$134 billion by 2025.
Historically, genomics has been a tool for the discovery domains of life sciences and biopharma R&D, but now there are larger clinical genomics projects evolving, embracing medical diagnostics and clinical testing [1]. The expansion of these areas requires a greater need for scientists, clinicians and regulators to come together from these different domains to address common issues, such as data reproducibility and standardisation. Alignment of standards between discovery and the regulated domains of the industry would create the potential for the same datasets and analyses to be used across the discovery and the clinical R&D frameworks. This would allow data analysed in discovery to be included in CDx regulatory filings, rather than industry being required to repeat clinical studies to generate analyses suitable for regulatory filing for biomarkers. In future, retrospectively, we might be able to rescue drugs that have failed in clinical trials, if appropriate CDx could be identified that recognise particular subpopulations that either positively or negatively respond to the drug. The growth of large genome and biobanking initiatives also offers the opportunity to stratify patient populations and to identify candidate conditions for repurposing existing drugs, as well as candidates to participate in new drug trials with defined genotypes and/or phenotypes. The deployment of CDx is an essential condition in the development of precision medicine. This was the thinking that led to the initiation of a community of interest at the Pistoia Alliance [3], and this meeting was organised to bring together experts from research, clinical and technical sectors, drug industry, hospitals, notified bodies and regulatory agencies.

The biopharmaceutical industry perspective
There has been a continuous decline in the number of drugs obtaining marketing authorisation per billion dollars invested since the 1950s [4,5]. The current probability of success of a drug reaching the market from initial research is~3%. However, drugs with supporting genetic information have more than twice the chance of success than those without [6][7][8]. Currently,~50% of all marketed drugs have some genetic evidence supporting them. It is expected that 10% of all known drugs will have associated genetic predictors of efficacy. It is not unknown for pharmaceutical companies routinely to perform pharmacogenomic testing in development but, for many disease areas, there can be challenges with statistical power caused by insufficiently large populations in the datasets.
Genotyping by array costs ~US$65 per sample, whereas genotyping whole genomes is~10x more expensive, but for some studies it is much more informative. So, it is important to use the right technology to address the question that is being asked. Whole genome data, although more expensive, provides additional data, such as the effects of noncoding sequence variation and, potentially, copy number variation. The UK Biobank [9] is generating data on samples from 500 000 individuals, including a wide range of clinical and imaging data, and this generates over a trillion new data points per year. GSK, Regeneron and other companies are funding exome and genome sequencing of these samples too [10,11]. One example quoted showed that a task that had previously taken >18 months and >1000 emails to achieve could now be achieved in 1 day using the UK Biobank data. The UK Biobank contains~1000 loss of function variants [12] in genes that have the potential to predict probable actions of drugs targeting these genes. However, there are key issues around consent and the sharing of the results of genomic analyses with the contributing participants of the genomic data. UK Biobank has adopted the position that it would not share analysis results with the participants. Other groups are looking at mechanisms to share results but there are still legal issues and important ethical issues that need to be addressed and are exacerbated by the ever-increasing use of global clinical studies. These organisations include the Global Alliance for Genomics and Health (GA4GH) [13], professional organisations such as the American College of Medical Geneticists and Genomicists (ACMG) [14], the British Society for Genetic Medicine (BSGM) [15] the 100 000 Genomes Projects [16], the EU [17] and many other national and regional bodies [18].
The genetics testing quality perspective The European Molecular Genetics Quality Network (EMQN) [19] was modelled on UK NEQAS [20] and was launched in 1997 with EU FP4 funding. For external quality assessment (EQA), EMQN users register to participate and are provided with a biological sample, or samples (unknown to the user), which the user processes and analyses using their own systems, and then returns their results to EMQN. The EMQN then compares the user's results with their internal consensus-based standards and provides feedback to the user. This feedback includes the user's performance and an anonymised summary of the results from all participants in the form of graphs showing ranges and quartiles for each metric measured. Currently, there are no universal bestpractice guidelines for clinical implementation of NGS and, although participation in the EMQN is not compulsory, regulators and notified bodies do take note of EQA performance and if a laboratory is performing poorly its testing activity can be suspended. The EQA scheme for NGS has been designed to be platform-agnostic and can be run on samples from different disease areas ranging from single genes to whole genomes.
In 2013, the pilot EQA had 24 labs participating, and by 2017 this had increased to 260 labs for germline testing and 100 labs for somatic testing. For germline testing, 70% of the data was generated on Illumina platforms, whereas for somatic testing 40% was carried out on LifeTech platforms. In 2017, 87% of the data submitted was mapped using Burroughs-Wheeler alignment (BWA) [21] and 64% used Genome Analysis Toolkit (GATK) [22] for variant calling. Users should be aware that both of these tools were originally developed for research purposes and have subsequently been adapted for clinical testing. For germline data, 92% of submissions achieved >80% sensitivity and 94% achieved >80% precision for the human reference genome version GRCh37. For somatic data, 65% achieved >55% sensitivity and 94% achieved >80% precision for GRCh37. Although the test results were only shared with the lab being assessed, the summary data could be shared, and a list of participants is available, so a potential customer of a testing lab could find out whether that testing lab has been tested and could ask that provider to share their results. One of the test samples was a Personal Genome Project (PGP) individual and the consensus data obtained will be provided back to the PGP Consortium [23].
A key challenge is the lack of a harmonised standard for the Variant Call Format (VCF) [24] files used to report variants. VCF currently provides a framework format, but fields within it can be customised. It was reviewed how the EQA could be used to monitor or assess NGS pro-cesses. Because the EMQN services could be applied to any NGS test lab, the EMQN could be used by any commercial instrument and/or kit manufacturers to 'qualify' their products. EQA systems are invaluable and it was recommended that manufacturers should collect these data. However, it was noted that, although labs in the UK needed to take part in a quality process such as EQA (a mandatory requirement of ISO15489 accreditation), this was not the case in all countries.

The regulatory perspective
The European Union In Vitro Diagnostic Medical Devices Regulation (EU IVDR) [25] entered into force on 26 May 2017 and transitional activities will occur until the date of application of the EU IVDR on 26 May 2022. By that point, existing CE-marked products will need to have become CE marked under the new framework barring an additional 2-years' grace period for those devices already certified with notified bodies under the IVD Directive. There are three key performance indicators that make up part of an IVD application.
Scientific validityhow good is the evidence for the association between the biomarker and the clinical condition or physiological state? Analytical performancehow good is the IVD at detecting the biomarker? Clinical performancehow good is the IVD at predicting which patients are likely to respond to the corresponding medicinal product? Every IVD will require a new performance evaluation but not all will require a new performance study. There are some basic requirements for all IVD performance studies, and some additional requirements will apply to performance studies in different circumstances (e.g., for surgically invasive sample-taking, interventional studies, studies that create additional risks to subjects, studies that involve people from specific groups and companion diagnostics). Figure 1 provides a high-level schematic demonstrating the alignment between the development of a therapeutic and the development of its CDx.
The IVD regulations will embrace those devices used in clinical trials to stratify patients. One of the additional requirements will be to notify the competent authority and (unless using only left-over samples) the competent authority will then assess the application before the start of the study. The processes for the assessment of an IVD performance study can summarised in five steps: application -> verification -> assessment -> plan trial -> performance study reporting.
A CDx registration could in principle be for a particular analysis pipeline or a tool for processing patient data. When studying rare diseases, samples can be very scarce and, as such, sample sizes can be small. Currently, there are no defined requirements for sample size; however, the sample size must be appropriate for the study undertaken, taking into account the intended use of the device and performance requirements for the device.

The notified body perspective
It is important to be clear whether the proposed product is an IVD device or a testing service. Most CDx fall under EU 2017-746 IVDR Annex VIII Rule 3 [26]. As class C devices they would require notified body certification. Notified bodies are designated by an EU Competent Authority to perform conformity assessments. These are based on the evidence and the conclusions provided to ascertain whether a device conforms to the relevant requirements. As of April 2018, no notified bodies had been approved by competent authorities to perform conformity assessments under the new EU IVDR. Quality management systems (EU IVDR Annex IX) provide assurance that appropriate processes are in place to enable CE certification. It is important that CDx manufacturers fully engage with their notified bodies to ensure that the new regulations can be addressed effectively and in a timely manner. Figure 2 shows the interdependency of the three key performance indicators: clinical performance, analytical performance and scientific validity.  Metrics assessed by notified bodies as part of the approval process for companion diagnostics.
www.drugdiscoverytoday.com 3 Once a CDx is certified, there is an ongoing requirement for its performance to be notified, including: incident reporting, post-market surveillance including periodic safety update reports, maintaining clinical evidence and postmarket performance follow-up (PMPF). For class C and D devices, updates to the summary of safety and performance are required to be submitted at least annually.
At least 3 months of QMS data are required before a CDx manufacturer should apply to schedule a notified body onsite QMS assessment. Notified bodies are not permitted to provide advice but they can provide training. Some important and useful quality standards already exist. Medical

The CDx development and services perspective
It is essential to have effective and resilient quality management systems in place. Analytical validation standards take time to develop. There are some key activities that include: design controls, assay development, assay software, reagent and control manufacture and regulatory interaction. The intended use of an NGS-based assay should drive the selection of the assay platform and technology. It is the entire process (the 'wet' and the 'dry' parts) that comprise the CDx and laboratory and bioinformatics pipelines are the subject of regulatory scrutiny. Figure 3 shows a CDx development pipelineagain aligned with key therapeutic development stages. There are some NGS assay validation challenges and some initiatives have been established in an attempt to address them. For example, the Next-generation Sequencing: Standardization of Clinical Testing (Nex-stoCT) Workgroup established by the CDC [33,34] was set up to address gaps in metrics and processes for test validation and quality control, and this has led to further quality system standards such as the College of American Pathologist (CAP) All Common and Molecular Pathology Checklist [35] and the Wadsworth Center's CLEP NGS Guidelines [36], which incorporate NGS-specific requirements for assay validation and laboratory performance. Quality assurance needs to be maintained and monitored throughout the preanalytical, analytical and post-analytical phases of testing. For CDx, all the bioinformatics processes must be locked, validated and controlled.
In a recent FDA fact sheet on tumour profiling NGS tests [37], three levels of biomarkers were defined: (i) companion diagnostics; (ii) cancer mutations with evidence of clinical significance; and (iii) cancer mutations with potential clinical significance. The high complexity of NGS pipelines and the results generated cause challenges.
Assay validationthe need for assay performance to meet the regulatory requirements of a clinical trial assay (CTA) and subsequently as a CDx. Selection of a suitable approach for target capture and the sequencing platform. Bioinformatics analysis and reporting. Change controlrevalidation of pipeline after modifications or upgrades. Long-term storage and retrieval of data.

Concluding remarks
The rapidly developing science and technology underpinning developments in CDx will cause some interesting challenges for all the involved stakeholders. For example, could cloud-based,   How might one establish performance standards for VCF files, and for the variety of annotation resources used to annotate the variants in the VCF files? Will the arrival of new sequencing technologies bring new formats of sequence data (e.g., FAST5) and the need for new QC methods too? What might be the best way of managing the versioning and provenance of data sources? There needs to be a consensus-driven standards effort including cross-industry input and feedback. Standards need to be agreed for data, reference genomes, reference materials and quality management systems; and such standards should be defined globally rather than just nationally. There is significant alignment of standards across EU countries and the introduction of the new EU IVDR regulations should bring further harmonisation. Nevertheless, different information is required to obtain a CE mark in the EU from the information required to obtain an Investigational Device Exemption (IDE) in the USA. The FDA is looking for industry assistance to codevelop standards. The industry needs a consensus view on how to create regulatory-compliant versions of great public research bioinformatics tools such that they could receive CE marking. There is an increased need for the relevant stakeholders to work more closely together. Furthermore, there needs to be enhanced clarity around definitions, for example does a genome sequence equate to one analyte or to three-billion analytes?
There is a need to ensure there is sufficient bioinformatics expertise available to the regulatory agencies, notified bodies and indeed patient organisations. Is there potential for the MHRA and EQA to collaborate to approve containerised software solutions that would provide modular tools in a pre-approved format for end users? Can the reference material from EMQN/NIBSC [42] and agencies abroad such as NIST [43] be generated with a common specification? Technical working groups responsible for writing standards might usefully engage with the relevant trade organisations such as the British In Vitro Diagnostics Association [44] in the UK and perhaps, more broadly, MedTech Europe [45]. Perhaps a pre-competitive collaboration, open to all, and committed to lowering barriers to innovation in life sciences R&D and healthcare, could facilitate a community of interest for the relevant stakeholders such that progress might be made to: define standards for data formats (e.g., VCF for regulatory applications), reference genomes, reference materials and quality management systems; define ways to standardise analytical processes (e.g., containerised tools or workflows) for regulatory applications; identify commonality and differences between research and clinical data analysis standards, with the goal of aligning them as much as possible with minimal increased effort.
The Pistoia Alliance has a founding mandate to encourage its member organisations to work together in the precompetitive space. Among these member organisations it has a large number of pharmaceutical, biotechnology and technology providers. As such, it provides one example of a qualified entity needed to bring together all the different stakeholders that need to work together, including existing initiatives with shared interests such as GA4GH [13], pre-cisionFDA [46], EMQN [19], among others.

Conflicts of interest
John Wise is a consultant to the Pistoia Alliance which has funded the coordination of this multiauthor paper. Mike Furness was a consultant working for the Pistoia Alliance when this paper was drafted.