International External Quality Assessment of Molecular Detection of Rift Valley Fever Virus

Rift Valley fever (RVF) is a viral zoonosis that primarily affects animals resulting in considerable economic losses due to death and abortions among infected livestock. RVF also affects humans with clinical symptoms ranging from an influenza-like illness to a hemorrhagic fever. Over the past years, RVF virus (RVFV) has caused severe outbreaks in livestock and humans throughout Africa and regions of the world previously regarded as free of the virus. This situation prompts the need to evaluate the diagnostic capacity and performance of laboratories worldwide. Diagnostic methods for RVFV detection include virus isolation, antigen and antibody detection methods, and nucleic acid amplification techniques. Molecular methods such as reverse-transcriptase polymerase chain reaction and other newly developed techniques allow for a rapid and accurate detection of RVFV. This study aims to assess the efficiency and accurateness of RVFV molecular diagnostic methods used by expert laboratories worldwide. Thirty expert laboratories from 16 countries received a panel of 14 samples which included RVFV preparations representing several genetic lineages, a specificity control and negative controls. In this study we present the results of the first international external quality assessment (EQA) for the molecular diagnosis of RVF. Optimal results were reported by 64% of the analyses, 21% of the analyses achieved acceptable results and 15% of the results revealed that there is need for improvement. Evenly good performances were achieved by specific protocols which can therefore be recommended as an accurate molecular protocol for the diagnosis of RVF. Other protocols showed uneven performances revealing the need for improved optimization and standardization of these protocols.


Introduction
Rift Valley fever (RVF) is a mosquito-borne viral zoonosis that primarily affects animals but also has the capacity to infect humans. An epizootic of RVF is usually first indicated by a wave of unexplained abortions as infected pregnant livestock abort virtually 100% of fetuses. The disease is less fatal to humans as most human infections are asymptomatic and when clinical symptoms appear they are in majority influenza-like. Nevertheless, some cases may develop a severe RVF disease with variable clinical signs. More severe cases occur in 2% of the RVF cases and fall into three categories: liver necrosis with hemorrhaging, retinitis with visual impairment and meningoencephalitis [1,2].
The causative agent of RVF, the RVF virus (RVFV), is a negative-stranded RNA virus, a member of the genus Phlebovirus of the Bunyaviridae family. The number of identified viral lineages of RVFV has increased from 3 in an early analysis [3] to 7 in a 2007 study [4], and in the most recent report 15 distinct genetic groups were reported [5]. Phylogenetic analysis shows that the virus emerged in the mid-19 th century, but it was first identified in 1930 during an outbreak of abortions and deaths among sheep in the Rift Valley region of Kenya. In 1977-78, several millions of people were infected and more than 600 died during a severe epidemic in Egypt [6]. Since then, the geographical distribution of the virus has widely spread and now includes most countries of the African continent as well as Madagascar and the Arabian Peninsula. During the past five years, outbreaks have been reported in Kenya [7], Somalia, Tanzania [8], Sudan [9], Mayotte [10], Madagascar [11], Swaziland, South Africa and Mauritania [12,13] Another important concern is the increasing number of human fatalities during the most recent outbreaks [14].
The emergence or re-emergence of RVFV activity is periodic and associated with exceptionally heavy rainfalls which allow massive breeding of flood-water Aedes mosquitoes with the capacity for transovarial transmission [15] and other competent vectors such as Anopheles and Culex species [9]. These mosquitoes may initiate outbreaks among livestock, particularly breeds of cattle and sheep. The virus can be transmitted to humans by mosquito bite or by contact with infected tissues of domestic and wildlife ruminants. The sudden onset of large numbers of abortions and fatalities in RVFV affected livestock, resulting in the virus spread to humans can greatly strain public health and veterinary infrastructures.
Unavailability of effective antiviral drugs and commercial vaccines for human or animal use outside endemic countries, including the US and Europe, and the recent spread of RVFV beyond its usual boundaries has resulted in increased international demand for qualified diagnostic tools for a rapid and accurate diagnosis of RVF.
Diagnostic methods for RVFV detection include virus isolation [16], antigen [17,18] and antibody detection methods [19][20][21] and nucleic acid amplification techniques. Isolation procedures are expensive, time-consuming and require high biocontainment facilities. Serological methods such as antigen or antibodydetection enzyme immunoassays (EIA) require several samples and often lack sensitivity. Therefore, considerable efforts have been made to develop molecular methods which allow a rapid, accessible and accurate detection of RVFV. The use of direct diagnostic methods such as molecular methods, can detect the disease during the acute phase of the infection thus allowing efficient patient management, avoiding nosocomial cases and providing rapid outbreak response. Highly sensitive nucleic acid detection methods have been developed including polymerase chain reaction (PCR) assays such as reverse-transcriptase PCR (RT-PCR) [22], real-time RT-PCR (qRT-PCR) [23][24][25] and more recently real-time reverse-transcription loop-mediated isothermal amplification (RT-LAMP) [26] and recombinase polymerase amplification assays (RPA) [27].
The performance of the different techniques applied for molecular diagnosis of RVFV may vary between laboratories. External quality assessment (EQA) studies to assess the quality of RVFV molecular diagnostics have not been performed until now. The EQA study allows the participating laboratories to monitor the quality of current diagnosis, identify possible weaknesses of particular diagnostic methods and evaluate their capacity for surveillance activities. Therefore the first EQA study for the molecular diagnosis of RVFV was organized by the European Network for Diagnostics of 'Imported' Viral Diseases (ENIVD) (http://www.enivd.org) in 2012. Using the results of this study, the ENIVD can also provide support and advice to all laboratories performing RVFV molecular diagnosis.

Call for participation
A total of 33 laboratories involved in diagnostics of RVF infections were invited to participate in this study. Invitees were selected from the register of ENIVD members, national/regional reference laboratories for RVF or vector-borne diseases as well as on the basis of their contributions to the literature relevant to this topic. The participation to the study was open and free of charge and included publication of the results in a comparative and anonymous manner. This EQA was coordinated by the ENIVD following comparable procedures used during previous studies performed by the network [28,29].

Specimen preparation
A proficiency test panel of 14 samples was prepared which included inactivated and stable RVFV preparations generated from Vero E6 cell culture supernatants of different RVFV genetic lineages and origin. Viral cell supernatants were inactivated by heating for 1 h at 60uC and gamma irradiation (25 kilogray) to assure their non-infectivity. A serum sample spiked with Toscana virus, another phlebovirus, was included as a specificity control as well as two negative controls. The RVFV positive samples selected for this EQA panel are detailed in Table 1. Two dilutions of sample Tambul/Egypt/1994 and 5 dilutions of sample F057/ Kenya/2007 were obtained by serial 10-fold dilutions and included in the panel for sensitivity testing.
All virus material used for the preparation of the EQA panel was obtained from cell culture and not from clinical samples of infected patients. Therefore, there is no requirement for any ethical statement in this study.
All samples were diluted with fresh thawed human plasma previously confirmed as negative for RVFV. Aliquots of 100 ml were number-coded, freeze dried for 24 h (Christ, AlphaI-5, Hanau, Germany) and stored at 4uC until dispatched.

Validation and dispatch of the panel sets
Before dispatching the panels, 3 different sets of EQA samples were tested and validated by 2 expert laboratories. For validation, the samples were resuspended in 100 ml of water and the RNA extracted using the QIAamp viral RNA minikit (Qiagen, Hilden, Germany). The number of RVFV genome copies present in these samples was determined by qRT-PCR.
Panel samples were shipped by regular post at ambient temperature. We requested participant laboratories to resuspend the samples in 100 ml of water and to analyze the material as serum samples for nucleic acid detection of RVFV following their routine protocols. The EQA panels were distributed to participants with documentation including full instructions and an evaluation form to fill in their results. Participants were also asked to report information on the adopted protocol, the type of RVFV strain and the number of genome copies in each sample when possible as well as any problems encountered concerning the shipment or the packaging of the samples.

Evaluation of the results
To guarantee anonymous participation, an individual numerical identification code was assigned to the results reported by each laboratory. This number was followed by a letter (a, b, c) when distinguishable data sets of results based on different methods were sent.
The results were scored in reflection of analytical sensitivity and specificity as in previous EQA studies performed by the ENIVD [29,30]. We assigned one point for correct positive or negative

Author Summary
Rift Valley fever (RVF) is a zoonotic viral disease posing an increasing threat to animals and humans worldwide. Recent severe outbreaks of the disease in animal and human populations in endemic regions and outside the disease's traditional geographic boundaries necessitate the need for evaluating the diagnostic performance of RVF expert laboratories. Molecular methods are increasingly used for a rapid and accurate detection of viral nucleic acid. In this study we present the results of the first international external quality assessment (EQA) for the molecular diagnosis of RVF. Such EQA studies allow participating laboratories to monitor the quality and identify possible weaknesses of current diagnostic methods. Participants to this RVF EQA were 30 expert laboratories from 16 different countries worldwide. The study demonstrated that optimal results could be achieved by the majority of laboratories. Specific protocols showed evenly good performances and can therefore be recommended to all expert laboratories. However, other methods showed uneven performances suggesting the need for improved optimization and standardization of these protocols.  A total of 39 datasets were received including 5 double sets from laboratories using 2 methods (lab #6, 7, 21, 27 and 28) and 2 triple sets from lab #5 and #14. Methods used by the same laboratory could differ from the type of technique, the protocol used for a specific technique or the type of instrument used for a specific protocol.
Performances varied among laboratories and scores ranged from 7 to the maximum value of 14. Optimal results were reported by 64% (n = 25) of the analyses; 21% (n = 8) of the analyses achieved acceptable results due to the inability to detect one positive sample, and 15% (n = 6) revealed several false negative and/or one or more false positive results indicating that there is still need for improvement (Table 2 and 3).
Further information on the protocol applied for each diagnostic test was requested from the participants. Concerning the 34 sets of results obtained by qRT-PCR methods, 31 referred to published protocols and 3 used in-house protocols. Among the diagnostic test results referring to published protocols, 16        is specified in Table 2 and 3 as well as the references corresponding to each protocol. Performance comparisons between each type of technique are limited as some methods are not sufficiently represented (e.g. n = 1 for RPA and RT-LAMP). For this reason a statistical analysis could not reach any significant conclusion. Nevertheless some general observations can be clearly delineated from the results. The 32 datasets obtained by using qRT-PCR only, were all highly scored with 13 or 14 points. Nevertheless it is interesting to notice that the 2 datasets obtained with qRT-PCR combined with nested RT-PCR scored 11 and 12 points demonstrating lower performances than qRT-PCR alone. Laboratory #22 lost 2 points due to 2 false positives generated by nested RT-PCR and thus would have demonstrated optimal performances if considering the results of the qRT-PCR only.
The maximum score of 14 was also obtained by the only set of results reporting the use of RPA technology. On the other hand, none of the methods involving nested RT-PCR achieved an optimal performance and the corresponding scores ranged from 7 to 13. The dataset obtained with RT-LAMP showed a score of 11 mostly because of difficulties in detecting the less concentrated samples.
The sensitivity of the different diagnostic methods can be assessed by comparing the testing results of the serial dilutions of RVFV-Kenya/2007 (samples #2, #9, #12, #4, and #14) and RVFV-Egypt/1994 (samples #5 and #13). Observing the percentage of correct results for each serial dilution, we observed that all samples achieved a percentage over 90% except for the less concentrated sample of RVFV-Egypt/1994 (sample #13) which presented a percentage of correct results of 79% (Table 2 and 3). Additionally, the most concentrated dilution of RVFV-Kenya/ 2007 (sample #2) was the only sample of the serial dilution samples to be detected by all techniques. This indicates a correlation between increased dilution of the sample and low sensitivity in RNA detection as expected.
Nevertheless, decreased sensitivity does not appear to be the main reason for reporting false negatives. In fact, 38% of false negatives (9 out of 24) may be attributed to decreased sensitivity as these are corresponding to the false negatives obtained for the sample with the lowest genome copy number (sample #13). The rest of false negatives results were distributed equivalently among the other samples, except for sample #2 (highly concentrated dilution of RVFV-Kenya/2007) and sample #6 (RVFV-Zimbabwe/1979) which presented 100% of correct results. Furthermore, it is interesting to note that several datasets (#9, 15, 21b and 29) revealed false negative results at lower dilutions of a serial dilution but reported a positive correct result at a higher dilution for the same strain. Such false negative results cannot be attributed to sensitivity nor strain specificity issues but rather to the lack of reproducibility and consistency of the employed test procedure. This is a clear indication of the need to improve test procedures by applying standardized protocols and accurate testing procedures.
When comparing test results obtained for the different RVFV strains, we observed even performances indicating an overall good specificity as the different techniques can detect several strain types. However, some individual set of results indicated a potential lack of specificity concerning methods such as the qRT-PCR protocol from Garcia et al. [23] which was not able to detect both dilutions of the RVFV-Egypt/1994 strain, and the nested RT-PCR protocol of Sanchez-Seco et al. [34] which was not able to detect the South-Africa/1981 strain and the Central-African-Republic/1985 strain (Table 3). Nonetheless this information should be interpreted with precaution as only one laboratory reported the use of each of these protocols.
We can also have indications on the specificity of each diagnostic method by analyzing the testing results of the two negative controls (samples #7 and #8). Altogether, 8% of the datasets (3 out of 39) reported false positives. Two of these 3 datasets originated from nested RT-PCR techniques, one reported to use of the protocol of Sall et al. [22] and the other the protocol of Sanchez-Seco et al. [34]. The third false positive was engendered by qRT-PCR and indicated a low viral load (Table 3).
In order to evaluate practice in viral load determination, participants were requested to provide the number of copies of RVFV genome detected in positive samples. Over 73% of the laboratories (22 out of 30) reported quantitative results and are reported in bold in Table 3  In order to assess capacity and experience in sequencing and strain typing, participants were requested to specify the strain or genotype of the RVFV detected in each positive sample. Only 13% (4 out of 30) of the laboratories reported specifications on the strain type and only one data set reported this information for all positive samples (data not shown).

Discussion
RVF reference laboratories responded keenly to this EQA study (91% response rate), including laboratories situated in RVFV endemic countries such as South Africa and Saudi Arabia. Nonetheless, there is still a need to encourage more laboratories situated in RVF-endemic areas to participate in quality assurance programs. In fact, the increasing amplitude of this disease in Africa necessitates the rapid recognition of RVF outbreaks and implementation of effective control measures in order to prevent uncontrolled and wider spread of the virus.
Most of the laboratories (93%, 28 out of 30) reported the use of qRT-PCR techniques allowing a rapid detection as well as quantification of the virus genome. This confirms that the use of qRT-PCR has remarkably expanded although it requires expensive equipment. All datasets obtained by qRT-PCR only were scored with 13 or 14 points indicating an evenly high performance of all qRT-PCR procedures performed by the different laboratories.
Protocols  [31] as well as all in-house qRT-PCR protocols (dataset #6b, #10 and #24) have demonstrated the capacity of providing optimal performances indicating a good specificity and sensitivity for these techniques. The sets of results obtained by applying the qRT-PCR protocols of Mweango et al. 2012 [35], Garcia et al. 2001 [23] and Busquets et al. 2010 [32] did not achieve optimal performances (scores 13, 11 and 13 respectively) but these techniques are not sufficiently represented to conclude on their overall performances.
Information on the viral load of RVFV in human samples can be very useful to monitor the progress of clinical manifestations and to study the pathogenesis of RVFV. Interestingly, not all laboratories employing qRT-PCR techniques have reported quantified results and most of them (64%) reported the results as cycle thereshold (Ct) values and not the number of genome copies. This indicates that most laboratories do not resort to RVFV standards while performing qRT-PCR although such standards would allow them to quantify viral genome in each sample without performing any additional assay. Accordingly to the results of this EQA as well as previous EQA studies, there is still room for improvement concerning viral load determination [29,30].
The most widely used technique after qRT-PCR was nested RT-PCR with 5 laboratories which referred to 2 different protocols [22,34]. Nested RT-PCR performances varied greatly compared to qRT-PCR with scores ranging from 7 to 13 thus never reaching optimal performances. The dataset #14c obtained a score of 13 with the protocol of Sall et al. 2002 [22] because it could not detect the highest dilution of the RVFV-Egypt/1994 strain indicating a slightly low sensitivity just as observed for some of the qRT-PCR methods. Nevertheless other datasets referring to nested RT-PCR (#9 and #22) also reported false positive results indicating a lack of specificity of these procedures with both nested RT-PCR protocols [22,34].
It is interesting to notice the appearance of newly developed techniques which are suitable for rapid field diagnostics such as RT-LAMP developed in 2009 [26] and RPA technology developed in 2012 [27]. No general conclusion can be achieved concerning the performances of these two techniques as they both have been performed by only one laboratory. However RPA has shown optimal results for this EQA demonstrating equivalent sensitivity and specificity to the qRT-PCR techniques (dataset #27b).
On the other hand, RT-LAMP results indicated difficulties in detecting RVFV genome in the less concentrated samples of the panel (sample #4, #13 and #14). These results suggest some limitations in test sensitivity. However, very high test sensitivity is not essential for field diagnostics in an outbreak situation where most diagnosed patients are in the acute phase of the disease and are expected to present a high viremia.
Three laboratories have provided different sets of results which referred to the same technique and protocol but using different instruments (datasets #5b/c, #14a/b and #28 a/b). These datasets provided all optimal results by using two different instruments except for dataset #14 which reported a slightly lower sensitivity using the SmartCycler System from Cepheid (#14b, 13 points) compared to the 7500 Real-Time PCR System from Applied Biosystems (#14a, 14 points). However, this difference cannot be attributed with certainty to the use of a different instrument as result variability can also arise from a lack of repeatability of the procedure.
Only a few participants provided complete or partial information regarding strain typing (13%, 4 out of 30). However, correct results without strain or genetic lineage specification are satisfactory in the context of laboratory diagnosis. Nonetheless, RVFV strain typing is relevant for surveillance activities in order to monitor which strains are circulating in RVFV-endemic areas and what type of clinical manifestations are associated with these strains.
Comparing the results of this EQA panel to previous EQA studies [29,30,36], we observe a higher concordance in terms of performance within laboratories using the same type of diagnostic method. In fact, all qRT-PCR techniques demonstrated an overall good performance with scores ranging from 13 to 14. On the other hand, nested RT-PCR methods have shown a common need for improvement in terms of test sensitivity and/or specificity.
Nevertheless, variations in performance between laboratories using the same method were noted. The reason for such variations is difficult to establish but can be minimized by standardizing procedures, including controls and testing conditions.
In order to ensure optimal performances for RVFV molecular diagnosis in expert laboratories, we recommend conducting EQA studies on a regular basis. Future EQA studies should include a wide range of RVFV isolates with limiting concentrations to assess as precisely as possible the diagnostic performances of various molecular protocols in different reference laboratories.