Contrived Materials and a Data Set for the Evaluation of Liquid Biopsy Tests

A Blood Profiling Atlas in Cancer (BLOODPAC) Community Study Kyle M. Hernandez,*y Kelli S. Bramlett,z Phaedra Agius,x Jonathan Baden,{ Ru Cao,z Omoshile Clement,k Adam S. Corner,** Jonathan Craft,yy Dennis A. Dean, II,zz Jonathan R. Dry,x Kristina Grigaityte,x Robert L. Grossman,*xx{{ James Hicks,kk Nikki Higa,kk Timothy R. Holzer,*** Jeffrey Jensen,yyy Donald J. Johann,zzz Sigrid Katz,xxx Anand Kolatkar,kk Jennifer L. Keynton,{{{ Jerry S.H. Lee,kk,kkk Dianna Maar,** Jean-Francois Martini,{{ Christopher G. Meyer,y Peter C. Roberts,{{ Matt Ryder,yy Lea Salvatore,xx Jeoffrey J. Schageman,z Stella Somiari,**** Daniel Stetson,yyyy Mark Stern,{ Liya Xu,kk and Lauren C. Leimanzzzz

of the collaborative effort was to develop a data resource, the BLOODPAC Data Commons, now available to the liquid-biopsy community for further study. This resource can be used to support independent evaluations of results, data extension through data integration and new studies, and retrospective evaluation of data collection. (J Mol Diagn 2023, 25: 143e155; https://doi.org/10.1016/j.jmoldx.2022. 12.003) Circulating tumor (ct)-DNA holds great promise in early cancer detection; the identification of therapeutic targets, especially when tissue is not available; the evaluation of residual disease after curative-intent treatment; therapeutic monitoring; as well as resistance mapping and recurrence surveillance [US Food and Drug Administration (FDA), https://www.fda.gov/drugs/resources-informationapproved-drugs/cobas-egfr-mutation-test-v2; and Foundation Medicine, https://www.foundationmedicine.com/pressreleases/af7bb7df-2dcf-411f-bc7d-ebb8ab90d788, both last accessed January 12, 2023]. 1e4 For example, early work by Diehl et al 1 demonstrated that ctDNA could be used to monitor tumor dynamics in colorectal cancer patients undergoing surgery or chemotherapy, thus revealing its potential as a clinically useful biomarker. The oncology field has utilized multiple methods of detecting ctDNA, such as Droplet Digital PCR (ddPCR) and, more recently, massively parallel next-generation sequencing (NGS). ctDNA is present at very low levels against a background of wild-type (WT) cell-free (cf)-DNA that is derived from various origins. The FDA has approved the use of several ctDNA tests, including a qPCR test for use in the identification of epidermal growth factor receptor gene (EGFR) mutations in nonesmall-cell lung cancer patients, 2,3 as well as two NGS-based tests for use in detecting various genomic aberrations for therapeutic indications across a variety of solid tumors (FDA, https:// www.fda.gov/drugs/resources-information-approved-drugs/ cobas-egfr-mutation-test-v2; and Foundation Medicine, https://www.foundationmedicine.com/press-releases/af7bb7df-2dcf-411f-bc7d-ebb8ab90d788, both last accessed January 12, 2023). As novel ctDNA-based methods are advanced toward use in clinical practice to help inform decisions beyond targeted-therapy selection and tumor profiling in late-stage patients, rigorous demonstration of analytical performance continues to be essential in supporting the suitability of tests for their intended uses. Crucial to the adoption of liquid-biopsy methods is the demonstration of performance characteristics such as precision, accuracy, and reproducibility. 5e9 There are increasing efforts to develop standards, including those by the Foundation for the National Institutes of Health (FNIH) to identify contrived materials in support of future experiments, and by the FDA Sequencing Quality Control Phase 2 Working Group to develop synthetic internal standards for ctDNA measurements. 6,10 Different ctDNA tests have been shown to exhibit differential performance for replicate testing of patientderived materials 7 ; discordant results in previously published studies could have been attributable to decreased detection, particularly of mutations at low VAFs, as well as varying robustness of detection in difficult sequence contexts such as GC-rich and repetitive regions. Williams et al 6 showed the difficulty in detecting copy number variations and translocations in different assays when using reference standards. Although those variant types were outside of the scope of the current study, Williams et al 6 also showed decreased detection of singlenucleotide variants (SNVs) at low VAFs. The findings from those studies highlight the need for standardized, accessible, and efficient methods of comparing ctDNA tests and, ideally, benchmarking; however, patient-derived materials are limited in quantity, may not contain a number or type of alterations sufficient for thorough performance characterization, and cannot be easily accessed by all members of the ctDNA-testing community at large.
To address these limitations, a community collaboration across Blood Profiling Atlas in Cancer (BLOODPAC) members was designed with the explicit goal of increasing the quality and consistency of ctDNA analysis through multisite testing of commercially available reference materials to yield a data set that can serve as the foundation for additional and objective inter-laboratory comparisons. BLOODPAC brings together over 60 stakeholders across government, industry, nonprofit, and academic institutions to focus on translational efforts that aim to benefit cancer patients. These efforts include establishing publiceprivate partnerships, 11 deploying the BLOODPAC Data Commons (a data resource now available to the liquid-biopsy community for further study), 12 and conducting translational studies 7,13,14 such as the JFDI (Just Freaking Do It) project, a multicenter study of ctDNA commercially available reference material.
This report presents the results from the JFDI project. Participating independent laboratories used standard ctDNA testing methods and proprietary evaluation procedures to assess the performance of two manufacturers' commercially available reference materialsdeither nucleic acid suspended in Tris-EDTA buffer, or diluted into a synthetic plasma matrixdused to simulate patient-derived plasma specimens requiring extraction. Mutations were present in a multiplex format across a range of VAFs (0.1% to 5.0%). The study design included measurements of intra-and crossparticipant concordance, sensitivity, and specificity. Participating laboratories utilized different sequencing platforms, library-preparation methods, and analysis software to mimic real-world inter-laboratory comparisons which, it is proposed, could be facilitated using a foundational data set ( Figure 1).

Preparation of Reference Material
This study utilized commercially available contrived reference materials: the Multiplex I cfDNA Reference Standard Set (Horizon Discovery, Cambridge, UK) and Seraseq ctDNA Complete Mutation Mix (SeraCare Life Sciences Inc., Milford, MA). The Multiplex I cfDNA Reference Standard Set, as purified cfDNA reference or in synthetic matrix, is a pack of four vials, each containing 400 ng (20 mL) of cfDNA in Tris-EDTA (10 mmol/L Tris-HCl, 1 mmol/L EDTA), pH 8.0 at a concentration of 20 ng/mL. The cfDNA is manufactured by fragmenting genomic DNA derived from a proprietary cell-line blend containing eight variants [six SNVs, one deletion, and one insertion] at 0.1%, 1.0%, and 5.0% VAF each, as well as a 0% VAF (WT) for specificity assessment (Table 1). A mean cfDNA fragment length of 160 bp AE10% (144 to 176 bp) material was confirmed by automated electrophoresis using the TapeStation system and the D1000 DNA ScreenTape assay (Agilent Technologies, Santa Clara, CA). The VAFs of target variants were confirmed by ddPCR (Bio-Rad Laboratories, Pleasanton, CA). A final cfDNA concentration of 20 ng/mL AE10% (17.0 to 23.0 ng/mL) was confirmed with the Qubit Double- Figure 1 Overview of the multi-participant contrived reference materials study design. Each participant selected a single library-preparation method (hybrid capture or amplicon) and a sequencing platform [Ion Torrent (Thermo Fisher Scientific, Waltham, MA) or Illumina (San Diego, CA)] or Droplet Digital PCR to process commercially available contrived reference materials from two providers across multiple VAFs. Contrived materials were provided in two formatsdDNA in buffer and plasma mimeticdwith four overlapping variants (three affecting EGFR and one affecting KRAS ). Participants generated results on contrived materials in four VAF tiers that differed between the two providers. The primary end points of the study were mutation calls along with the observed VAFs. y Seraseq Circulating Tumor DNA Complete [SeraCare Life Sciences, Milford, MA; VAF tiers: WT, 0.1%, 0.5%, and 1.0%; included 19 variants). z Multiplex I Cell-Free DNA Reference Standard Set (Horizon Therapeutics, Dublin, Ireland; VAF tiers: WT, 0.1%, 1.0%, and 5.0%; included 8 variants).

Contrived Liquid Biopsy Assessment
The Journal of Molecular Diagnosticsjmdjournal.org Stranded (ds)-DNA BR assay (InvitroGen, Carlsbad, CA) prior to aliquoting. Given that Horizon's reference material is derived from cell lines, whole-exome sequencing data revealed many additional variants across the genome that were present in the background of the eight ddPCRverified variants.
The Seraseq ctDNA Complete Mutation Mix and plasma reference materials provided for the evaluation study consisted of 25 cancer-relevant DNA variants spiked into a background of genomic DNA derived from cell line GM24385 (Coriell Institute for Medical Research, Camden, NJ) at VAFs of approximately 1.0%, 0.5%, 0.1%, and 0% (WT) ( Table 2). Of the 25 variants, 6 were structural variants and were beyond the scope of this effort, thus they were removed from all analyses. The fragmented DNA was prepared at a length between 160 and 180 bp using a SeraCareproprietary process. The Seraseq ctDNA Complete Reference Material was formulated in a synthetic plasma matrix with a DNA concentration of approximately 25 ng/mL in 5 mL (total extractable DNA of approximately 125 ng) and stored refrigerated. The Seraseq ctDNA Complete Mutation Mix was formulated in a 0.1Â Tris-EDTAebased buffer (10 mmol/L potassium chloride, 1 mmol/L Tris, and 0.1 mmol/L EDTA, pH 8.0) at a concentration of approximately 15 ng/ mL, and stored at À20 C. Digital (d)-PCR measurements were performed prior to DNA fragmentation.
Each commercial reference material supplier covered a different set of variants (8 and 19), with 3 EGFR variants and 1 KRAS variant shared between both sets. Due to the imbalanced nature of this study, no direct comparisons were made between reference materials.

Multi-Participant Sequencing and Processing
Contrived materials from each vendor, VAF, and DNA format [purified cfDNA in buffer (hereafter, DNA), and cfDNA diluted into synthetic plasma (hereafter, plasma)] were provided to nine independent-laboratory participants. Each participant processed samples using their standard target-enrichment method (amplicon or hybridization capture) and sequencing platform (Illumina or Ion Torrent) or Droplet Digital PCR platform (catalog number QX200; Bio-Rad Laboratories), in triplicate (Table 3 and Figure 1) at a fixed amount of total input DNA (25 ng); participant total input DNA ranged from 20 to 25 ng. Many participants used existing amplicon or hybridization capture panels that covered many more targets than those specifically present in the contrived materials and that sometimes did not cover all of those present in the contrived materials (Supplemental Table 1).
Participants applied their own in-house bioinformatics workflows for each replicate and provided a final set of observed VAFs, undetected variants, and variants not covered by their sequencing panels. Details about the bioinformatics pipelines employed by the participants are provided in Participant Specific Materials. The submitted variant calls and observed VAFs were used to determine whether a variant was classified as detected or missed. Variants occurring in regions not covered by participants' panels were considered as not applicable.

Participant-Specific Materials
University of Arkansas for Medical Sciences/Fluxion Biosciences The series of control samples described here was used as input into the Spotlight 59 assay (amplicon based; Fluxion Biosciences, Oakland, CA) to generate targeted libraries measuring key oncogenic driver and resistance mutations.
Sample preparation, including nucleic acid extraction, DNA library building, and validation, was performed in accordance with the Spotlight 59 assay. The libraries were then sequenced on a NextSeq 500 system (high-output flow cells, 150 bp PE; Illumina, San Diego, CA). A total of 36 NGS experiments were performed, as summarized in this manuscript (University of Arkansas for Medical Sciences/ Fluxion Phase II Sample Set). The analysis strategy was based on the bioinformatics software package ERASE-Seq v2.4b (Fluxion), which contains a variant caller that does  Bio-Rad Laboratories ddPCR involves partitioning a PCR reaction mix into 20,000 uniform-size droplets, thermal cycling to end-point fluorescence, and then reading the fluorescence of each individual droplet. Using a locus (SNV)-specific assay to amplify the DNA targets of interest resulted in a fluorescent signal derived from the droplets that contained either the WT or variant DNA target, while no signal was detected from those that did not contain target DNA. Thus, each droplet was counted and a number of copies of WT versus variant and VAFs in the sample were reported. An assay mix containing 25 ng of control DNA, ddPCR Supermix for Probes (no deoxyuridine triphosphate; Bio-Rad Laboratories, Hercules, CA), and Mutation Detection Variants 20 to 25 are structural variants and were not included in this study, which was focused only on single-nucleotide variants (SNVs) and small insertions/deletions (indels

Contrived Liquid Biopsy Assessment
The Journal of Molecular Diagnosticsjmdjournal.org assay (Bio-Rad Laboratories) at final concentrations of 900 nmol/L per primer and 250 nmol/L per probe in a 22 mL final volume was prepared. The specific assays used included: i) BRAF V600E (dHsaCP2000027), ii) KRAS G12D (dHsaCP2000001, dHsaCP2500596), and iii) EGFR pE746-A750del (dHsaCP2000039). A volume of 20 mL of assay mix and 70 mL of ddPCR droplet oil (Bio-Rad Laboratories) were transferred onto a QX200 Droplet Generator cartridge then loaded into the QX200 Droplet Generator (Bio-Rad Laboratories). Vacuum was applied, pulling individual samples and oil through a flowfocusing junction to produce approximately 20,000 water-inoil droplets. A volume of 40 mL of the oil and sample droplet emulsions were then transferred into a 96-well plate and thermocycled in a standard thermocycler (Bio-Rad Laboratories) at 95 C for 10 minutes, 94 C for 10 minutes, and 55 C for 1 minute (repeated 40 times), and 98 C for 10 minutes. The plate was then transferred to a QX200 Droplet Reader and analyzed by QuantaSoft software version 1.7 (Bio-Rad Laboratories).
VAF percentage was calculated and submitted to BLOODPAC for inclusion in the JFDI project data and subsequent statistical analysis.

Bristol Myers Squibb
The QIAamp Circulating Nucleic Acid Kit (Qiagen, Hilden, Germany) was used for the extraction of the plasma mimetic samples, with a final elution volume of 50 mL. Both extracted plasma samples and cfDNA samples were quantified in the range of 75 to 250 bp with the High-Sensitivity Large Fragment kit (catalog number DNF-493; Agilent Technologies) for the 5300 Fragment Analyzer system per the manufacturer's recommendations to determine 25 ng input into library preparation.
An enrichment-based library-preparation method was used. End-repair and A-tailing of cfDNA samples were performed using customized reagents. Unique Molecular Identifier DNA Index Anchors from the TruSight Oncology 500 High-Throughput (TSO500 HT) assay kit (Illumina) were then ligated. Libraries were amplified using primers from the same kit. For enrichment, a single 2-hour hybridization using TSO500 HT reagents, with modifications, was followed by two capture washes and PCR using Bio-Rad C1000 thermal cycler to amplify enriched libraries. Cleanedup libraries were normalized using bead-based normalization from the TSO500 HT kit according to the manufacturer's instructions. Normalized libraries were pooled as 24-plexes and sequenced on NovaSeq 6000 S4 flow cells (Illumina). Data were analyzed using a customized analysis pipeline.
Eli Lilly and Co. For samples in synthetic plasma matrix, cfDNA was isolated using the QIAamp cfDNA/RNA kit (Qiagen). Targeted sequencing libraries were prepared using the Oncomine Pan-Cancer Cell-Free Assay (Thermo Fisher Scientific, Waltham, MA) according to the manufacturer's protocol with 25 ng of contrived cDNA as input. Libraries were quantified using the Qubit High-Sensitivity dsDNA Assay (Thermo Fisher Scientific) and library quality was assessed on the TapeStation 4200 system (high sensitivity; Agilent Technologies). Libraries were diluted to 100 pmol/L and pooled. Templating was performed using the Ion 540 kit on the Ion Chef and then sequenced on the Ion Torrent S5XL system (Thermo Fisher Scientific). Reads were analyzed using Torrent Suite variant caller plugin software version 5.12 (Thermo Fisher Scientific). All sequencing output was reproduced three times in independent runs.

Illumina
The plasma mimetic samples were extracted using the QIAamp Circulating Nucleic Acid Kit (Qiagen) according to the manufacturer's instructions. The cfDNA was eluted in 50 mL of the provided buffer.
The concentrations of the extracted plasma mimetic and the cfDNA were measured on the Fragment Analyzer system using the High Sensitivity Large Fragment kit (catalog number DNF-493; Agilent Technologies) according to the manufacturer's instructions. A concentration in the range of 75 to 250 ng was used to determine the input into the TruSight Oncology (TSO)-500 ctDNA assay kit (Illumina). The manufacturer of the TSO500 ctDNA assay recommends 30 ng of DNA input for a DNA size of 75 to 250 bp measured on the Fragment Analyzer. However, for this study, targeted libraries were generated from 25 ng of input to be consistent with study design parameters. The TSO500 ctDNA assay is an enrichment-based library-preparation kit that enriches 523 genes and enables tumor profiling. The libraries were sequenced on the NovaSeq 6000 sequencer (Illumina). Libraries were sequenced as a 24-plex on an S4 flow cell in the XP workflow. The DRAGEN TSO500 ctDNA software (Illumina) was used for primary data analysis and complex variant calling.

Pfizer
For Pfizer's reference standard analysis of cfDNA, the MagMAX cfDNA isolation kit (InvitroGen) extraction method was used. The extraction protocol was automated on the KingFisher Flex system (Thermo Fisher Scientific), with 2 mL of plasma input per sample. The library-preparation method was a modified version of the SureSelect XT procedure (Agilent). Briefly, cfDNA samples were ligated with sample indexes containing random unique molecular identifier sequences and amplified using the KAPA HyperPrep kit (Roche Holding AG, Basel, Switzerland). Amplified samples were enriched for a 134-gene whole-exon panel using the SureSelect XT capture hybridization procedure. Enriched libraries were then washed, amplified, and quantified using quantitative real-time RT-PCR. Final libraries were sequenced on an NovaSeq S4 Flow Cell (Illumina). Data were analyzed using a customized analysis pipeline involving Tri-Nucleotide Error Reducer error suppression to increase the specificity of variant calling while maintaining high sensitivity of detection. jmdjournal.org -The Journal of Molecular Diagnostics

Sysmex
The contrived samples were used as input into a customized Safe-Seq assay (Illumina) to generate targeted NGS libraries focused on key oncogenic drivers and resistance mutations. The libraries were then sequenced on an NextSeq 550 sequencer (Illumina). Libraries were sequenced at a depth of at least 2Â the number of input genome equivalents (GEs) resulting in a mean read depth of >100,000Â for each amplicon (for 8000 GE per sample; read depth scales with input amount). The library preparation is an amplicon-based method that incorporates unique molecular tags to identify individual molecules of input DNA, allowing for high sensitivity and specificity of calling rare variants while identifying and removing workflow errors from variant calling. A proprietary data analysis pipeline specifically designed and developed for Safe-Seq was used for variant calling.
For reference materials supplied in a synthetic plasma matrix, DNA was extracted from the plasma-like matrix using the QIAamp Circulating Nucleic Acid Kit (Qiagen). A total of 8000 GEs of contrived DNA were used as input for library preparation, corresponding to the amount of DNA in 25.0 ng of actual cfDNA. The number of GEs per microliter was confirmed by LINE1 qPCR prior to library preparation.

Thermo Fisher Scientific
The data contributed by Thermo Fisher Scientific were derived from four sets of contrived samples from Horizon Discovery and SeraCare. For the contrived material that is in a plasma background, 2 mL (Horizon) or 4 mL (SeraCare) of the control material was purified manually using the MagMAX cell-free total nucleic acid isolation kit (Thermo Fisher Scientific) following the user's guide and instructions for a higher concentration of cfDNA. All of the control materials were measured for cfDNA quantity using a Qubit fluorometer (Thermo Fisher Scientific) and 25 ng of the contrived material was used as input into the library preparation. Sequencing libraries were generated with the Oncomine Pan-Cancer Cell-Free Assay (Thermo Fisher Scientific) to generate targeted libraries measuring key oncogenic drivers and resistance mutations. Libraries were generated manually following the instructions in the user's guide. The library method is an amplicon-based method that incorporates unique molecular tags to identify individual molecules of input DNA that are captured and amplified into a targeted library for sequencing on the Ion Torrent sequencing platforms, allowing for high sensitivity and specificity of calling rare variants while identifying and removing workflow errors from variant calling. The resultant libraries were then sequenced on an Ion S5 XL System (Thermo Fisher Scientific) using the Ion Chef Instrument (Thermo Fisher Scientific) for template preparation. Libraries were sequenced as a 4-plex on the 540 chip (Horizon) or as a 6-plex on the 550 chip (Seraseq) with a target of 12 to 15 million reads per library, allowing >25,000Â coverage of each targeted amplicon or gene region (mean of 40,000Â coverage for the entire panel). Torrent Suite software version 5.6 and Ion Reporter software version 5.6 (Thermo Fisher Scientific) were used for primary data analysis and complex variant calling, using analysis workflows optimized for this library-preparation kit.

University of Southern California
Targeted sequencing libraries were prepared using the Oncomine Pan-Cancer Cell-Free Assay (Thermo Fisher Scientific) according to the manufacturer's protocol, with 20 ng of contrived cfDNA as input. Libraries were quantified using the Qubit High-Sensitivity dsDNA Assay (Thermo Fisher Scientific) and Ion Library TaqMan Quantitation Kit (Thermo Fisher Scientific), and library quality was assessed using the Agilent 2100 Bioanalyzer and High-Sensitivity DNA assay (Agilent Technologies). Libraries were diluted to the recommended 100 pmol/L, pooled for templating with the Ion 550 kit and Ion Chef instrument (Thermo Fisher Scientific), and sequenced on the Ion S5 system (Thermo Fisher Scientific). Sequencing data were analyzed using Torrent Suite software version 5.12.1 and Ion Reporter software version 5.6. The Oncomine TagSeq Pan-Cancer Liquid Biopsy-w2.1-Single Sample workflow (Thermo Fisher Scientific) was used with default parameters for variant calling.

Participant Sensitivity, Specificity, and Concordance Rate
The sensitivity of each unique participant replicate was calculated as TP/(TP þ FN), where TP (true positive) is the number of variants classified as detected, and FN (false negative) is the number of variants classified as missed, excluding those not covered by the participant's panel. Specificity was estimated as TN/(TN þ FP) using the WT reference sample, where TN (true negative) is the number of times that base positions corresponding to variants included in the reference material were correctly identified as WT or NMD (no mutation detected), and FP (false positive) is the number of times that variants present in the reference material were incorrectly detected in the WT tier; for each participant, only covered positions were considered. Concordance rates were estimated both within (intra-) and across (intra-) participants. To estimate intra-participant concordance rate, detection data were first grouped by participant, reference material format (DNA versus plasma), contrived material source (SeraCare versus Horizon), and expected VAF level. Within each group, the detection status of each pairwise replicate combination was compared, excluding variants not covered by that participant's panel. These comparisons were then used to calculate the concordance rate as N c /mean (N a , N b ), where Nc is the number of concordant detected variants between two replicates, and N a is the number of detected variants in replicate A and N b is the number of detected variants in replicate B. For cross-participant concordance, a similar approach was

Contrived Liquid Biopsy Assessment
The Journal of Molecular Diagnosticsjmdjournal.org applied, but instead of comparing replicates within participants, each replicate from a given participant was compared with replicates from all other participants, limiting to only the intersection of variants covered by both participants.

BLOODPAC Multi-Participant Contrived Reference Material Study
A community-based effort was organized to demonstrate the utility of contrived reference materials in inter-laboratory testing of ctDNA assays and analyses across a range of VAFs. Commercially available contrived reference materials from two vendors were distributed to multiple participants within the BLOODPAC consortium (Figure 1). A total of nine independent participants received reference materials across VAF tiers (Horizon: WT, 0.1%, 1.0%, 5.0%; SeraCare: WT, 0.1%, 0.5%, 1.0%) sourced from synthetic plasma DNA or ctDNA. Of these, eight participants used NGS assays while one used an orthogonal technology, ddPCR. Across the eight NGS-based participants, five used Illumina sequencing platforms and three used Ion Torrent (Thermo Fisher Scientific) platforms. Of the five Illumina users, two performed target enrichment using an amplicon-based approach while three used hybridization capture; all three Ion Torrent users prepared sequencing libraries via an amplicon-based method.
This study was focused on 19 variants from the SeraCare materials and 8 variants from the Horizon materials, of which 4 variants were shared across both sets of reference materials. To reduce the logistical impact on participants, each participant utilized existing panels, resulting in different coverage across the study (Supplemental Tables S1 and S2). The participant utilizing ddPCR covered a small number of variants, of which only one was common to the subset included in this study and was excluded from the results except where noted (a total of eight participants after removal).

Observed VAF in NGS Assays
The distribution of observed VAFs across participants using NGS-based assays showed strong concordance between expected and observed VAFs with both vendors and reference-material formats). Figure 2 shows reported results, with values missing for some participants given that not all participants provided results on each experimental factor (DNA and plasma) and variant. To compare observed VAFs between DNA and plasma samples, the data set was first reduced to participants who tested both formats and variants that were covered by at least three of these participants (Figure 3). These results showed similar VAFs observed between the DNA and plasma formats. In the Horizon reference materials, NRAS p.A59T showed consistent differences in observed VAFs between the DNA and plasma formats across VAF tiers (Figure 3), likely arising from natural variability within manufacturing of the reference materials (Horizon, personal communication, August 9, 2022).

Performance of NGS Assays
Performance was measured by first estimating the participant-level sensitivity and specificity of each replicate ( Figure 4). The number of participants providing data differed across reference materials, DNA formats, and expected VAF (Supplemental Table S2). In general, specificity was high across all participants, regardless of DNA format. Although sensitivity varied between participants, there was an overall pattern of reduced sensitivity at the 0.1% VAF tier. The single overlapping variant (KRAS p.G12D) covered in the orthogonal ddPCR platform exhibited high sensitivity (100% at all tiers; VAF, 0.1% to 5.0%) coupled with low specificity (approximately 33%), in contrast to the majority of participants that used the NGS platform ( Figure 4). Although the sample size was far too low to make generalizable inferences, these findings illustrate a need to better understand platform-specific limitations.
Next, both inter-and intra-participant concordance rates of detected variants were estimated as a proxy for reproducibility ( Figure 5). In general, concordance was reduced at the 0.1% VAF tiers and inter-participant concordance rates were often lower than intra-participant concordance rates.

Discussion
The stated goals of BLOODPAC and the Cancer Moonshot (https://obamawhitehouse.archives.gov/sites/default/files/ docs/fact_sheet_final_moonshot.pdf, last accessed January 12, 2023) include breaking down barriers between research groups and encouraging the sharing of data and results across both academic and commercial entities. In line with those goals, this study was born out of an extended discussion during a general BLOODPAC meeting in which the topic was "How To Compare Results and Determine Best Practices for Mutation Analysis of Cell-Free DNA." The participants included voices from largescale developers of companion diagnostics tests, to biotech companies developing new genome-wide methodologies, plus data scientists and academic genomics experts. The topics were wide ranging, including the difficulties jmdjournal.org -The Journal of Molecular Diagnostics associated with combining proprietary results into one publicly accessible database and the potentially competing interests of the members. Finally, one participant indicated, "Let's just do it, instead of talking, and see what happens"; thus, in the spirit of cooperation typified by BLOODPAC, the current study was born.
As the four original participant groups grew to nine over the course of a year, the theme of the study became clearer: to serve as a showcase for the utility of contrived reference materials. Although multiple methods were being compared, the idea was not to have a bake-off to determine the best test method, but instead to evaluate broadly available reference materials and share the resulting data with the liquid-biopsy community. Variant calls from sequencing data were submitted by each of the participating groups to the BLOOD-PAC Data Commons (https://data.bloodpac.org, last accessed August 25, 2022), where they are publicly available. The data analysis presented here was performed by various volunteers within the working group.
Perhaps the most important result for the general liquidbiopsy community from an operational point of view was the evaluation of four contrived cfDNA sample preparations manufactured by two of the most well-known standards providers, Horizon and SeraCare, through the use of diverse sequencing methods. These contrived preparations consist of sheared, well-characterized genomic DNA, to which DNA fragments containing a variety of variant alleles (SNPs and indels) have been added at predetermined allelic fractions (VAF) and offered as both cfDNA-in-buffer as well as plasma-mimetic formats. The limited overlap in variants included in the two contrived preparations prevented performing a strict statistical test, but on visual inspection of the data, the sensitivity and reproducibility of the assays used for mutation detection were highly consistent and comparably dependent on the VAF according to the dilutions supplied by the manufacturer. The conclusion of this study is that the sensitivity levels of the contrived preparations from SeraCare and Horizon are nearly

Contrived Liquid Biopsy Assessment
The Journal of Molecular Diagnosticsjmdjournal.org indistinguishable at higher VAF levels across DNA and plasma formats. Additionally, the findings from this study demonstrate that the contrived reference materials performed comparably across nine independent study sites, using three different technologies to assess cfDNA and using two different methodologies for sample enrichment. These results clearly support the use of these reference materials in the development and technical validation of liquid-biopsy workflows. The findings also confirmed the conclusions of Williams et al 6 : that the reference materials are a valuable resource in the validation of assays. These materials can help to one understand the accuracy, reproducibility, and repeatability of a method for evaluating variants from cfDNA when clinical samples containing specific variant types are very limited and are used to exhaustion quickly and without the capability for adequate replication to support rigorous statistical analysis.
Circulating cfDNA from cancer patients is a variable mixture of normal and tumor-derived genomic sequences. The reproducibility of variant allele detection depends directly on the actual number of variant molecules in the assay input, which in turn depends on the total amount of cfDNA available from the blood draw and the VAF. The tests for this study were performed using 25 ng of total fragmented DNA, which translates to roughly 7600 human haploid genome equivalents. Therefore, at a 1% dilution (1% VAF), a mean of 76 variant allele sequences per assay input would be expected, with minimal variability from vial to vial of control material. Detectability is more challenging at 0.1% VAF, where the maximum variant molecules input for testing is expected to be seven to eight. In practice, far fewer than seven or eight molecules are expected to be available for target enrichment due to cfDNA fragmentation. Although 0.1% is clearly in the theoretical range of detectability, random variation results in a greater number of stochastic false negatives at this low analyte level; additionally, other variables such as nonoptimal library conversion efficiency and mutations present in difficult sequence contexts can further negatively impact sensitivity. Due to the limitations of the study design, these potential sources of variation were not explored; however, other studies have explored these relationships in detail. 14 The findings from this study were comparable to those from a study hosted by the FNIH, 6 with a similar goal of understanding the effectiveness of contrived materials to advance liquid biopsy. Some major differences between the study discussed here and the FNIH study is on the reference standards evaluated. The present study reviewed cfDNA reference standards that are commercially available, whereas the FNIH study began with generating new reference materials that were unique to the FNIH study and not broadly commercially available at the time of this Figure 3 Observed mean AE SEM VAFs by variant, aggregated by the source of DNA material, VAF tier, and reference material. Only participants with data from both DNA and plasma sources and variants evenly covered by at least three participants across DNA sources were included. The WT VAF tier represents false-positive results, while all other VAF tiers represent true-positives. At the 0.5%, 1.0%, and 5.0% VAF tiers, observed VAF values were consistent between DNA and plasma; at the 0.1% tier, there was less consistency. A: Multiplex I Circulating Free DNA Reference Standard Set (Horizon Therapeutics, Dublin, Ireland). Here, NRAS:p.A59T exhibited a consistent difference between DNA and plasma sources across the VAF tiers. B: Seraseq Circulating Tumor DNA Complete (SeraCare Life Sciences, Milford, MA). jmdjournal.org -The Journal of Molecular Diagnostics publication. There was quite a bit of overlap in the participants of this JFDI BLOODPAC study and the FNIH study, and the results were shared and discussed between the two groups as the testing commenced. Lastly, the data from the FNIH study is not available to the public, whereas all of the variant-call data from the present study is publicly available through the BLOODPAC Data Commons.
The Cancer Moonshot initiative announced the formation of BLOODPAC in 2017, with the goal of unifying efforts among academic, government, biotechnology, diagnostic, and pharmaceutical entities in the liquid-biopsy space. The BLOODPAC members responded to this charge to collect, harmonize, and make data freely available through the BLOODPAC Data Commons, a cloud-based platform that utilizes the same framework as the National Cancer Institute's Cancer Research Data Commons to provide data storage and analysis support. The Commons currently hosts 73 studies and projects that include over 4400 cases. The availability of these data sets motivated studies aimed at informing the liquid-biopsy community both on best practices, including minimum technical data elements 13 and analytical variables, 7 and on available resources. 14 Some aspects of the experimental design included in this publication should be considered when evaluating the data. Processing (sequencing) or analysis procedures were not constrained, but instead study participants were asked to use the methods they had validated in their institutes for the evaluation of cell-free nucleic acid from plasma. Participants were free to use established data-collection processes and to use their optimized bioinformatics methodology for each technology. One of the important results of this study was the demonstration, under a data-sharing cooperative such as BLOODPAC, that commercial entities can collaborate productively. In addition, the independent evaluation performed here might generate similar results across centers and lay the groundwork for multicenter trial designs. The generated data set will support independent evaluation and extension that has the potential for generating new findings. The results support follow-up testing of clinical samples, where metrics similar to those presented here could be used to demonstrate functional equivalence of these commercially available reference materials with patient-derived specimens. This comparison may also benefit from further, more extensive orthogonal platform

Contrived Liquid Biopsy Assessment
The Journal of Molecular Diagnosticsjmdjournal.org testing, across digital PCR platforms and other potential methods.