Interlaboratory Comparison of the Pneumococcal Multiplex Opsonophagocytic Assays and Their Level of Agreement for Determination of Antibody Function in Pediatric Sera

When measuring a functional antibody response to pneumococcal immunization, it is imperative that a specific, reproducible, accurate, and standardized assay with acceptable inter- and intra-assay variation be advocated internationally to allow for meaningful comparison of results between laboratories. We report here the results of a collaboration between 3 international laboratories testing 30 pediatric samples against the 13 serotypes in Prevenar13.

Opsonophagocytic assays (OPAs) have been developed to evaluate pneumococcal vaccine immunogenicity by mimicking in vivo opsonophagocytosis.
Guidelines for the measurement of functional serotype-specific pneumococcal antibodies are detailed in the World Health Organization document Protocol for Multiplexed Opsonophagocytic Killing Assay (UAB-MOPA) for Antibodies against Streptococcus pneumoniae (5). However, laboratories use a variety of methods, often with minor modifications, which may result in varied results. It is therefore imperative that a specific, reproducible, accurate, and standardized assay with acceptable inter-and intra-assay variation be advocated internationally to allow for meaningful comparison of results between laboratories.
We report here a bridging exercise undertaken in 2014 and 2015 between three international pneumococcal research laboratories. For the purposes of this report, the laboratories have been deidentified. Each laboratory used its own standardized MOPA protocols and reagents to test the common pediatric serum sample panel. This is the first multilaboratory study comparing MOPAs with pediatric samples.

RESULTS
OPAs. Laboratories A and B provided results for all samples. Laboratory C was unable to report the results for one sample for serotype 1, nine samples for serotype 3, two samples for serotype 6A, one sample for serotype 18C, and three samples for serotype 19A due to the samples not passing their laboratory assay criteria of a maximum killing percentage of Յ40% or Ն70% (refer to Table S1 in the supplemental material). A negative titer (Ͻ8) was recorded as 4 and was included in all statistical analyses. The geometric mean opsonic index (GMOI) and 95% confidence intervals (CI) of results submitted by each laboratory are detailed in Table 1. Refer to Tables S2 to S4 for the GMOI and 95% CI for sample 1, sample 2, and sample 3.
Scatterplots and Lin's CCCs. Figure 1 displays representative scatterplots of the OIs for sample 1, sample 2, and sample 3 when each laboratory's results are plotted against the other. For the two serotypes in Prevnar (a 7-valent pneumococcal conjugate vaccine [PCV7]), serotypes 4 and 18C, agreement was good, and in general, the points cluster around the diagonal line of agreement. Lin's concordance correlation coefficient (CCC) supports this agreement; for serotype 4, in lab A and lab B, the CCC was 0.80, in lab A and lab C, the CCC was 0.80, in lab B and lab C, the CCC was 0.88 (also for serotype 18C), in lab A and lab C, the CCC was 0.95, in lab A and lab B, the CCC was 0.89, and in lab B and lab C, the CCC was 0.94 (Table 2).
For the two non-PCV7 serotypes, 3 and 7F, there was greater discordance. For serotype 3, 53% of samples tested by lab A were below the limit of detection, while this percentage was 19% or 20% for lab C or lab B, respectively. In the lab A paired scatterplots for serotype 3, this is clearly seen with the majority of matched titers from both lab B and lab C above the diagonal line of agreement. For a serotype 3 comparison between lab B and lab C, Lin's concordance data support a good concordance of 0.86; however, the scatterplot demonstrates that the OIs reported by lab B were higher overall. This was also seen for serotype 7F, with lab A reporting a negative titer for 33% (10 of 30) of samples and lab C reporting a negative titer for 7% (2 of 30) of samples. Lab B reported a positive titer for all 30 samples. Scatterplots for the remaining serotypes are shown in Fig. S1 to S3. Overall, Lin's concordance coefficients indicate a moderate-to-high agreement between lab A and lab C. The concordance between lab A and lab C for pooled OI results was 0.79 (95% CI, 0.75 to 0.83), while serotype-specific coefficients ranged between 0.42 (serotype 7F) and 0.98 (serotype 1). The agreement was serotype dependent, with highest agreement for serotypes 1, 5, and 18C and lowest agreement for serotypes 7F and 14.
The concordance between lab A and lab B for all serotypes combined was 0.65 (95% CI, 0.60 to 0.69), while serotype-specific coefficients ranged between 0.21 (serotype 7F) and 0.89 (serotype 18C). The agreement was also serotype dependent, with highest agreement for serotypes 4, 5, and 18C and lowest agreement for serotypes 7F and 9V.
The concordance between lab B and lab C for all serotypes combined was 0.88 (95% CI, 0.86 to 0.90), while serotype-specific coefficients ranged between 0.48 (serotype 7F) and 0.94 (serotype 18C). The agreement was serotype dependent, with highest agreement for serotypes 6A, 18C, and 4 and lowest agreement for serotypes 7F and 9V.
Figures S1 to S3 demonstrate scatterplots between the three laboratories, which show good agreement for most serotypes after injection of the 13-valent pneumococcal conjugate vaccine (PCV13), with the points clustering around the diagonal line. However, for serotypes 7F and 14, the agreement was weak, as shown by Lin's concordance coefficients (for serotype 7F, the lab A and lab C CCC was 0.42, for lab A and lab B, the CCC was 0.21, and for lab B and lab C, the CCC was 0.48; for serotype 14, the lab A and lab C CCC was 0.45, the lab A and lab B CCC was 0.29, and the lab B and lab C CCC was 0.60) ( Tables 3 to 5).
Opsonic index and serotype-specific IgG. At the time of sample 1, all children were aged 18 months and had received 2 or 3 infant doses of PCV7. The samples were taken 1 month after a reduced (20%) dose of 23-valent Pneumovax (23vPPV). The serotype-specific IgG geometric mean concentration (GMC) for all PCV7 serotypes was greater than 2.06 g/ml, and the three laboratories reported a positive titer (OI Ն 8) for all 7 serotypes. The serotype-specific IgG GMC and corresponding OI for the non-PCV7 serotypes were substantially lower (Table S2). For sample 2, the serotype-specific IgG GMC for 8 of 12 serotypes (6 of 7 PCV7 serotypes) had waned significantly (P Ͻ 0.05) and was reflected by lower OIs for the majority of serotypes, as measured by the three laboratories (Table S3).
One month after the PCV13 booster (sample 3), the serotype-specific IgG GMCs for all PCV13 serotypes had increased significantly (P Ͻ 0.03), and the OIs reported by the three laboratories were all positive (Ն8) for PCV7 serotypes as well as serotypes 6A, 7F, and 19A. The OIs were Ͻ8 for six samples, namely, five samples for serotype 1 and one sample for serotype 3 from lab A. OIs were also Ͻ8 for four samples for serotype 1 and two samples for serotype 5 from lab C and one sample for serotype 1 from lab B (Table S4).
ICCs. Intraclass correlation coefficients (ICCs) were used to examine agreement across all three laboratories ( Table 2). The ICCs ranged between 0.58 and 0.95 (median ICC ϭ 0.83), indicating moderate-to-high agreement between laboratories. Again, similar patterns of results emerged, with high ICC values for serotypes 18C, 6B, and 19F and lower ICCs for serotypes 7F and 14.
Comparison of samples with positive (titer, Ն8) and negative (titer, Ͻ8) OPA results between laboratories resulted in the following. All three laboratories recorded a negative titer as Ͻ8. Kappa coefficients were used to assess agreement in the classification of titers as positive or negative (Tables 3 to 5). We calculated overall kappa coefficients for all serotypes combined, as we were unable to compute serotypespecific kappa coefficients due to responses falling into too few categories. The overall agreement was 92.8% for lab B and lab C, with a kappa coefficient of 0.75. This indicates good agreement between lab B and lab C in terms of positive or negative classification. Agreement was slightly lower for comparisons between lab A and lab C (agreement ϭ 90.4%, kappa ϭ 0.73) and lower for comparisons between lab A and lab B (agreement ϭ 83.3%, kappa ϭ 0.51).

DISCUSSION
This paper describes the results from three independent laboratories measuring pneumococcal opsonophagocytic responses in pediatric sera following PCV7 immunization and in the same children before and after PCV13 immunization. For the results from vaccination studies to be compared internationally, clinicians, laboratories, and pharmaceutical companies must have confidence that the assays provide reproducible and meaningful results. Here we describe the results from three laboratories, Murdoch Children's Research Institute's Pneumococcal Immunology Laboratory (MCRI; Australia), the UCL Great Ormond Street Institute of Child Health (UCL; United Kingdom), and the   (6). In 2003, five laboratories performing OPAs took part in a multilaboratory evaluation (7). Each laboratory was provided with the same protocol, seven target pneumococcal strains (serotypes 4, 6B, 9V, 14, 18C, 19F, and 23F), two quality control sera, and 12 paired sera from adult donors who received one dose of 23vPPV. The report concluded that the assay could be done in multiple laboratories with a high degree of accuracy. Then in 2011, six laboratories (the original five plus one) assessed agreement using their own optimized protocols (8). Five of six laboratories assayed 24 reference specimens, and one laboratory assayed 19 unique samples. The sample profile initially included 3 samples from patients preimmunized with 23vPPV and 21 samples from patients postimmunized with 23vPPV, with three preimmunization samples removed from the analysis, as the majority of the assayed values were at or less than the minimum measurable titer. The final number of samples included in the analysis was 16 samples from healthy adults who had received 23vPPV. Five laboratories tested for at least 12 of the 13 serotypes in PCV13 (one laboratory failed to report titers for serotype 3), and one laboratory tested for the 7 serotypes in PCV7. The precision, accuracy, and concordance between the postimmunization samples included in the analysis demonstrated an acceptable agreement.
Our study is the first to perform an interlaboratory comparison using pediatric samples. This is important since evaluation of PCV immunogenicity is critical following immunization during infancy. The critical components of an OPA are the pneumococcal strain used, the preparation and encapsulation of the serotype, the source of the complement, heat inactivation of the sera, and the cultured phagocytes. The method differences between the laboratories were minimal (Table 6). Each laboratory used identical organisms from BEI Resources (with differences in the methods used for culturing the strains and subtle differences in the MOPA groupings), the same source for the baby rabbit complement, and the same phagocytic cells, with similar levels of differentiation and passaging steps. Therefore, the differences in the results obtained between the different laboratories are likely to be due to other factors, given the similar protocols used.
In the present study, lab B and lab C reported high OIs to nonvaccine serotypes. Song et al. (9) previously reported high OPA titers to serotype 7F in subjects who had received only PCV7. They concluded that the high titer might not be vaccine induced and may confer no protection in those individuals. Lin's concordance coefficient and intraclass correlations were used to assess agreement and accuracy between the laboratories, and kappa coefficients were used to assess agreement in the classification of titers as positive or negative. Subgroup statistics to compare pre-and postimmunization samples were inappropriate, as each group comprised only 10 samples, and lab C supplied OI results for fewer samples. However, examination of the supplementary figures demonstrates a good relationship between all postimmunization (PCV13) sam-  MOPA Comparison following Pneumococcal Immunization ples and those serotypes in the vaccine. This is in agreement with the findings of Juergens et al. (10), who reported a significant correlation of serotype-specific IgG levels and OPA results for infants postvaccination with PCV13. The agreement was not as strong, especially for the non-PCV7 serotypes, in samples taken at 12 months after PCV7 vaccination and in samples taken preimmunization with PCV13 more than 4 years later (sample 1 and sample 2). This may be due to the low-titer serotype-specific IgG. Lab B and lab C detected functional responses to non-PCV7 serotypes, whereas lab A reported negative titers in those samples. The majority of immunization trials report an OI as Ն8 (positive) or Ͻ8 (negative), with no reference to the absolute serotype-specific IgG concentration. As such, it is important that the laboratories performing an OPA are consistent about measuring at this Ն8 or Ͻ8 interface. Song et al. also noted that the functionality of anti-19F antibodies was at least 10-fold lower than the functionality of antibodies to the other serotypes in PCV7. At that time, they suggested that the cutoff of Ն8 may need to be altered for each serotype. New cutoff values for both serotype-specific IgG and OPAs were published by Andrews et al. in 2014 (11), and when these cutoff values were applied to the results from the current study, the agreement between the three laboratories was even stronger. Our study demonstrates different responses to individual serotypes, with the highest OIs reported to serotypes 6B and 9V and the lowest OIs to serotype 1, 3, and 4. It has been shown that the polysaccharide-bound antibody isotypes of IgG2 and IgG3, as well as IgM, can activate complement-mediated opsonization through the classical pathway. Thus, the measurement of serotype-specific IgG and OPAs do not always correlate. This is commonly seen in adult sera (9,12), where nonspecific IgG with low functional capacity, as well as IgM, may be present. However, in pediatric postimmunization sera, the correlation is generally much stronger (10,13,14). Low levels of nonspecific IgG with low functional capacity may explain the difference in OIs measured for non-PCV7 serotypes, described in the current study for those samples after a microdose of 23vPPV and 4 years later (sample 1 and sample 2). Minor differences in methods used by the three laboratories, including the ratios of pneumococci to HL-60 cells, the storage and viability of pneumococci, the speed of the rotating mixing platform, the serotype panels used, and HL-60 differentiation protocols, may all affect the final result. This study was designed to compare the results from three independent laboratories performing assays for measurement of functional responses to pneumococcal immunization. There was no stipulation with regard to the method used, which serotypes to include in the MOPA, the growth and number of the bacteria, the growth and number of HL-60 cells, or the relative acceptance criteria (sensitivity and lower limit of measurement and uncertainty). For our current study to have maximum impact, all of these factors should be investigated further to ensure that we have confidence in the methods used in our laboratories.
A recent report from an international collaboration assigned OIs to the pneumococcal reference serum of lot 007sp and an FDA OPA calibration serum, which means that, for the first time, laboratories conducting opsonophagocytosis assays now have a designated standard to run in each of their assays (15). This will help to improve the accuracy and standardization of these assays in future studies. Furthermore, the introduction of an annual quality assurance program would be a most worthwhile exercise for all laboratories performing these assays to ensure that immunogenicity results from ongoing vaccine studies are reported with confidence.
Conclusions. This is the first report of an interlaboratory comparison of opsonophagocytic responses in pediatric samples. The level of agreement between the MCRI, UCL, and THL laboratories depended on the timing of the sample collection and was good for vaccine serotypes, while agreement was lower for nonvaccine serotypes. Overall, this study demonstrates good agreement in OPA measurement in pediatric samples across three laboratories, and despite minor differences in protocol, the OI results from each laboratory may be compared with confidence.

MATERIALS AND METHODS
Participating laboratories. The Murdoch Children's Research Institute Pneumococcal Immunology Laboratory (MCRI), the WHO Pneumococcal Serology Reference Laboratory, UCL Great Ormond Street Institute of Child Health, United Kingdom (UCL) and the National Institute for Health and Welfare Vaccine Immunology Laboratory, Helsinki, Finland (THL). The laboratories have been randomly deidentified, and results are reported as from lab A, lab B, and lab C.
Samples. Thirty sera from a sample collection stored at the MCRI were separated into three equal aliquots and were sent on dry ice to the WHO Pneumococcal Serology Reference Laboratory, UCL Institute of Child Health, United Kingdom (UCL), and the National Institute for Health and Welfare, Helsinki, Finland (THL). The samples were from 10 children who were part of the Fiji Pneumococcal Project and the Fiji follow-up study. Full details of these studies have been published previously (16,17). Ethics approval was obtained from the Fiji National Research Ethics Review Committee (FNRERC) and the Royal Children's Hospital Human Research Ethics Committee (Melbourne, Australia). Three samples previously collected from each study participant were used.
Five children received 3 doses of Prevnar (PCV7) at 6, 10, and 14 weeks, and 5 children received 2 doses of PCV7 at 6 and 14 weeks. At 18 months of age, all 10 children received a 20% dose of 23-valent Pneumovax (23vPPV). Sample 1 was collected at 4 weeks after 23vPPV administration, at 18 months of age. As part of the Fiji study, the guardians of these children were contacted to permit the children to be part of a follow-up study when their ages were between 4.9 and 7.4 years, and the children received a further dose of the 13-valent pneumococcal conjugate vaccine (PCV13). Sample 2 was taken prior to the PCV13 booster, and sample 3 was taken 1 month after the immunization (Table 7).
Participating laboratories were asked to include the sera in their routine assays and perform opsonophagocytosis assays (OPAs) on the serum samples using their own method for all 13 serotypes in PCV13 (1, 3 Serotype-specific IgG. Serotype-specific IgG was measured in the Pneumococcal Immunology Laboratory at the MCRI using a previously described modification of the gold standard WHO enzymelinked immunosorbent assay (18).
OPAs. Each laboratory's method is based on the multiplex opsonophagocytic assays (MOPAs) and OPAs developed at the University of Alabama, Birmingham, AL (5). Minor method differences between the three laboratories are summarized in Table 6. In general, sera were heat inactivated at 56°C for 30 min prior to serial dilutions in a 96-well sterile microtiter plate containing Hanks' balanced salt solution with Mg 2ϩ , Ca 2ϩ , and gelatine (opsono buffer [OB]). Frozen stocks of pneumococci were thawed and washed with OB and diluted to 5 ϫ 10 4 CFU/serotype/ml. Standard bacterial dilutions were added to all wells, and the plate was incubated at room temperature for 30 min. At 30 min, 10 l of baby rabbit complement, thawed just prior to use, was added, followed by addition of 40 l of HL-60 cells (2 ϫ 10 7 cells/ml) to all test wells. A bacterial control (heat-inactivated fetal calf serum [FCS] in place of human serum and no complement) and a complement control (no serum) were included on all plates. Plates were placed on a horizontal shaker and incubated for 45 min at 37°C in 5% CO 2 . The reaction was stopped at 45 min by placing the plate on ice. A 10-l aliquot of this mixture was then spotted onto Todd-Hewitt broth-yeast extract (THYE; 0.5%) agar plates. After application of an overlay of THYE agar containing selective antibiotics (optochin, spectinomycin, streptomycin, or trimethoprim) and 2,3,5-triphenyltetrazolium chloride (TTC), the plates were incubated overnight at 37°C in 5% CO 2 .
TTC is recommended for the detection of microbial growth by means of TTC reduction and aids in the counting of colonies using an automatic colony counter. After overnight incubation, colonies on the plates were counted, and the results were expressed as opsonization indices (OIs), where the OI is defined as the interpolated dilution of serum that kills 50% of bacteria. OIs were determined using either an in-house (THL) software or Opsotiter 3 (license granted to Licensee by UABRF) (19).
Statistics. Serotype-specific IgG concentrations and OIs were reported using geometric means and 95% confidence intervals. Summary statistics are presented separately by serotype and laboratory. The OIs were classified as being positive or negative based on the current recommended cutoff value of Ͻ8 (negative) and Ն8 (positive). All negative titers were assigned a value of 4 and included in all statistical analyses. Lin's concordance correlation coefficients (20) were used to examine agreement in OIs between each pair of laboratories (i.e., lab A versus lab B, lab A versus lab C, and lab B versus lab C). Lin's concordance coefficient is a combined measure of accuracy (how well the data follow the 45°line of perfect agreement on a square scatterplot) and precision (how tightly the points cluster around the line). A value of 0 indicates no concordance, while a value of 1 indicates perfect concordance.
Intraclass correlation coefficients (ICCs) were used to assess agreement among the OIs across the three laboratories. The ICC values were estimated using a mixed-effects analysis of variance (ANOVA) model with a fixed effect for laboratory and a random effect for sample. The ICC is a ratio of the variance among samples to the total variance. ICC values can range between 0 and 1, with higher values representing higher correlations between measurements of the same sample.
To examine agreement in classification between the laboratories, we displayed the data using 2 by 2 contingency tables and calculated kappa coefficients, a chance-corrected measure of agreement between two categorical variables. A kappa coefficient of 0 indicates no agreement, while a value of 1 indicates perfect agreement (21). Landis and Koch (21) classification of the kappa coefficients states that 0.00 to 0.20 indicates slight agreement, 0.21 to 0.40 indicates fair agreement, 0.41 to 0.60 indicates moderate agreement, 0.61 to 0.80 indicates substantial agreement, and 0.81 to 1.00 indicates almost perfect agreement.
Due to nonnormality, OIs were log transformed, using a base 10 logarithmic transformation, prior to calculations of Lin's concordance coefficients and ICCs. All coefficients of agreement were calculated using Stata version 14.1, while figures were produced in Stata and GraphPad Prism. A P of Ͻ0.05 was considered statistically significant in all cases, unless stated otherwise.
Subgroup analyses. Post hoc analysis of geometric means and 95% confidence intervals for serotype-specific IgG and OIs were calculated for each blood collection time point to investigate differences between laboratories.