Abstract
Increased use of powerful PCR technology for the routine detection of pathogens has focused attention on the need for international validation and preparation of official non-commercial guidelines. Bacteria of epidemiological importance should be the prime focus, although a “validation infrastructure” once established could easily be adapted for PCR-based detection of viruses and parasites. The aim of standardization should be the widespread adoption of diagnostic PCR for routine pathogen testing. European experience provides the impetus for realization of this vision through preparation of quantitative reference DNA material and reagents, production of stringent protocols and tools for thermal cycler performance testing, uncomplicated sample preparation techniques, and extensive ring trials for assessment of the efficacy of selected matrix/pathogen detection protocols.
The strength of diagnostic PCR, as opposed to investigative PCR, is its ability to rapidly and reproducibly screen for negative samples, which allows more resources to be directed towards characterization and epidemiological tracking of positive isolates. In addition, for slow-growing pathogens, intracellular bacteria, viruses, and viable-but-non-culturable pathogens, PCR has opened up new detection possibilities (1).
While investigative PCR is mainly used ad hoc for a limited period by a few technicians and for a specific research project, diagnostic PCR has to perform reliably and consistently day after day in the hands of different staff and on different samples. The latter is of particular relevance for method validation, since the well-recognized inhibition of DNA polymerase by many different constituents of the sample matrix is the Achilles' heal of PCR (2). This is the main reason that – especially for PCR – validation and sample preparation should be seen as two sides of the same coin.
INTEGRATED APPROACH
Each element of diagnostic PCR could be dealt with separately, but a more integrated approach should take into consideration the three following issues: Sample-specific method development and validation, establishment of an internal quality assurance scheme, and, finally, participation in external proficiency testing ring trials, also called external quality assurance (EQA) programs (Fig. 1).
An integrated approach to establishment of diagnostic PCR.
EQA programs for nucleic acid-based diagnostics have not been widely implemented in clinical laboratories (3). This is partly due to the limited availability and/or difficulties in the shipment of clinical material (4), which has hampered the evaluation and standardization of tests. While internal and external quality assurance programs apply universally to any diagnostic test, a samples-specific approach is crucial to PCR. Why is that so? By looking at established culture-based methods for pathogen detection, it is perceived that the same “horizontal” culture protocol is usually recommended for all types of samples, regardless of the source and matrix, e.g. Salmonella enterica (5). This has led some workers to follow the same culture dogma when developing PCR for diagnostic purposes. It goes without saying that fecal samples have a very different composition than urine samples. But what makes this difference even more pronounced for PCR is the inhibitory effect specific for sample type. It is thus necessary to choose the sample type before evaluating a primer set for its selectivity.
SAMPLING AND SAMPLE PREPARATION
The performance of diagnostic PCR is limited in part by the presence of inhibitory substances, even in very small amounts. For example, PCR assays containing the widely used thermostable DNA polymerase from Thermus aquaticus, Taq DNA polymerase, are totally inhibited in the presence of as little as 0.004% (vol/vol) blood (6). Therefore, efficient sample processing procedures prior to PCR are needed to improve the test performance. A sample processing step has several aims: it should not only overcome PCR inhibition but also concentrate target nucleic acids/cells (in the case of subclinically infected samples) and, furthermore, turn the heterogeneous biological sample into a homogeneous PCR-compatible sample. The latter is of importance since the composition of certain matrices can show drastic batch-to-batch variations. This indicates the need for a PCR-compatible sample of comparable composition, independent of the variation in the original matrix.
In order to comply with the aforementioned requirements, different pre-PCR processing strategies have been described (7). There are the three steps in between a sample matrix to be tested and the amplification reaction: sampling, sample preparation, and preparation of the amplification mixture.
The sampling procedure has an impact on the downstream work. For example, the type of swabbing material used influences the concentration of the target microorganism recovered (8, 9). Depending on the sample type and the target pathogen, different aspects of sampling may be emphasized when choosing a suitable sampling method.
DIFFERENT GOALS BUT ONE PROTOCOL
If the goal is to remove PCR inhibitors only, other aims such as concentration of target nucleic acids/cells, turning heterogeneous biological samples into homogeneous PCR samples, and detection of viable (and not dead) cells have not yet been achieved. There are two steps left when this can be done. The first step and the easiest way to work towards these goals is adjustment of the amplification mixture. PCR inhibition can be reduced by the choice of an appropriate DNA polymerase and/or amplification facilitators that may resist PCR inhibitors, and thus maintain the robustness of diagnostic PCR in the presence of inhibitory substances (for review see (7)). In fact, the overall performance of diagnostic PCR, e.g. amplification efficiency and linear range of amplification, may be improved by the use of appropriate DNA polymerase and PCR reagents (9). Nonetheless, adjustment of the amplification mixture alone is rarely sufficient, which means that additional sample treatment prior to preparation of the amplification mixture is needed. Numerous sample preparation protocols have been developed (for reviews see (10)) and one or several can be selected depending on the sample type, sampling method and amplification mixture. Finally, depending on the choices made, the mentioned goals – as well as time and costs – are affected and will determine the overall success of the PCR strategy in detecting a target pathogen in a biological matrix. This can be illustrated by the selection of a fast cheap sampling method and sample treatment procedure, which might not recover sufficient target copies and will therefore demand a more sensitive PCR assay. It is thus important to follow the integrated approach by choosing a sample type, sampling method, sample treatment and PCR mixture, in combination with a PCR assay, and validate the whole system rather than just a part of it (11).
VALIDATION USING GOLD STANDARDS
This brings us back to the well-known dilemma of PCR vs traditional culture-based methods, where we are actually comparing “apples” with “oranges” (12). In PCR, we are amplifying DNA, while culture-based methods isolate live bacteria, in some cases leaving “stressed” infectious target bacteria behind. The stress can be due to initial antibiotic treatment before the sample is sent to the laboratory for testing. Many workers have addressed this issue by spike-in experiments that demonstrate a detection limit of one target bacterium in a 25 g sample, as required for environmental samples. However, the extrapolation of results from a test in spiked studies using fresh cultures of “healthy” inoculates to routine analysis can justifiably be questioned.
As a result, some confusion exists in the use of terminology during the course of validation (12). Recent international documents have provided useful guidelines for the correct use of terms (Table 1), and simple formulas for calculation of agreement between gold standard and PCR (Table 2) (13, 14).
| Validation | Results obtained by PCR should be comparable to those obtained by the reference method. |
| Qualitative PCR | The test response is either the presence or absence of PCR product (amplicon), detected either by observation or with equipment. |
| Quantitative PCR | The test response can be correlated with the DNA copy number of amplicon, related to the number of target microorganisms. |
| Detection limit (DL) | The smallest number of culturable target microorganisms necessary to create a PCR-positive response. |
| Selectivity | Measure of inclusivity of target strains (from a wide range of strains), and exclusivity (the lack of amplicon from a relevant range of closely related non-target strains). |
| Positive deviation (PD) | PCR-positive case when the reference method gives a negative result (false positive). |
| Negative deviation (ND) | PCR-negative case when the reference method gives a positive result (false negative). |
| Positive agreement (PA) | Sample positive by both PCR and the reference method. |
| Negative agreement (NA) | Sample negative by both PCR and the reference method. |
| Diagnostic accuracy (AC) | Degree of correspondence between the response obtained by PCR and the response obtained by the reference method on identical samples (AC=(PA+NA)/total number of samples). |
| Diagnostic sensitivity (SE) | Ability of PCR to detect the microorganism when it is detected by the reference method ((PA/N+) ×100). |
| Diagnostic specificity (SP) | Ability of PCR to not detect the microorganism when it is not detected by the reference method ((NA/N−) ×100). |
| Robustness | Reproducibility by other laboratories using different batches and brands of reagents and validated thermal cyclers and equipment. |
- N− is the total number of negative results with the reference method.N+ is the total number of positive results with the reference method.
| PCR response | Reference method positive (R+) | Reference method negative (R−) |
|---|---|---|
| Alternative method positive (A+) | +/+ Positive agreement (PA) | −/+ Positive deviation (PD) (R−/A+) |
| Alternative method negative (A−) | +/− Negative deviation (ND) (A−/R+) | −/− Negative agreement (NA) |
However, the robustness of a test is best challenged by data produced by diagnostic staff working in real-life situations on unselected clinical samples. Such real-life data would be very helpful for clinical diagnostic laboratories assessing PCR tests from the literature. It is the overall use and clinical performance of the test under field conditions that interests clinicians.
MEASUREMENT OF TEST VARIATION
To establish routine diagnostic PCR methods it is necessary to investigate specific parameters, e.g. specificity, detection limit, linearity, precision, etc. Mathematical and statistical models provide an indication of the reproducibility of PCR testing (15). The advantage of using these models is that interpretation of the results is objectively based. For example, the detection limit for a method should reflect the entire method, including sampling, sample preparation, nucleic acid amplification, and finally detection of PCR products. Unfortunately, most articles report the detection limit for the PCR assay without dealing with the pre-PCR steps. When studying the detection limit it should be borne in mind that PCR analysis employs only small sample volumes, e.g. 5 μl. If the detection limit is stated to be 1 CFU/ml, the probability of the cell being in the 5 μL PCR sample is very low, i.e. one positive out of 200 reactions, if no concentration step is included in the pre-PCR processing of the sample. To improve the reliability this limit should be associated with the probability of detecting the target DNA/cell at a certain concentration. Löfström et al. (16) have used a logistic regression model to accurately determine the probability of detecting small numbers of salmonellae in feed samples, in the presence of natural background flora (Fig. 2). From this model, the probability of detecting 1 CFU per 25 g of feed in soy samples was calculated and found to be 0.81.
The graph illustrates the detection probability at various DNA/cell concentrations. Experimental data obtained by plotting the concentration of DNA/cells against the observed relative frequencies of positive PCR detection may be used to generate a logistic regression model (15). The model describes the detection probability at various PCR template concentrations. Löfström et al. (16) have used this principle to accurately determine the PCR detection limit for salmonellae.
IMPORTANCE OF TEST CONTROLS
As recommended according to international standards (13), PCR cannot be given diagnostic status before it includes, as a minimum, an internal amplification control (IAC), a processing positive-control, a reagent control (blank) and a processing negative control (Table 3). The inclusion of IAC is of particular importance for diagnostic PCR (17), although careful consideration should be given to design of a proper IAC (18).
| Internal amplification control (IAC) | Containing chimeric non-relevant DNA added to master mixture. |
| Processing positive control (PPC) | Negative sample spiked with sufficient pathogen and processed throughout the entire protocol. |
| Processing negative control (PNC) | Negative sample spiked with sufficient closely related, but non-target, strain processed throughout the entire protocol. |
| Reagent control (blank) | Containing all reagents, but no nucleic acid apart from the primers. |
| Premises control | Tube containing the master mixture left open in the PCR set-up room to detect possible contaminating DNA in the environment (carried out at regular intervals as part of the quality assurance program). |
| Standard concentrations | 3 to 4 samples containing 10-fold dilution series of known number of target DNA copies in a range above the detection limit. |
False negatives are undesirable and can be damaging. They prevent us from focusing on a specific disease, bringing with them the extra costs and complications of continuing with unnecessary drugs and investigations while searching for a diagnosis. This damage extends from all test types, not just PCR. It would be desirable to include an internal control for culture methods; however, to the best of our knowledge, this is not technically possible.
In contrast to a positive control (external), an IAC is a non-target DNA sequence present in the very same sample tube, which is co-amplified simultaneously with the target sequence. In a PCR without an IAC, a negative response (no band or signal) could mean that there was no target sequence present in the reaction, but it could also mean that the reaction was inhibited. Where a positive control checks for errors, such as incorrect PCR mixture or poor DNA polymerase activity, an IAC will provide an appropriate way to ensure there is no inhibition in the actual tube due to inhibitory substances in the sample matrix or malfunctions of that part of the thermal cycler area (19). Conversely, in a PCR with an IAC, a control signal should always be produced even though there is no target sequence present. This can reveal failure of a PCR reaction.
STANDARDIZATION CAN FACILITATE QUALITY ASSURANCE
As a model for integrated validation and quality assurance of genetic tools, a European-level strategy was established by the FOOD-PCR project ((20); www.pcr.dk). Recognizing the need for standard PCRs in order to avoid excessive first-time validation in each end-user laboratory, European activities were initiated with the aim of validating and standardizing the use of diagnostic PCR for detection of bacterial pathogens in foods. The work continues to provide an example for standardization efforts in the molecular genetics field (www.medvetnet.net).
In addition to sample preparation methods, production of reference DNA material (21), preparation of a thermal cycler validation guideline and tools (19), and performance of PCR ring trials were included. Another important area was automated detection, including semi-quantitative real-time PCRs (22).
Amongst the important outcomes of the project were guidelines and a biochemical kit for validation of thermal cyclers (SureCycle from www.congen.de), a simple method for purifying DNA from bacterial cultures, production of reference DNA material, workshops organized for end users, and preparation of standardized guidelines in collaboration with the European Committee on Standardization (CEN), Working Group 6.
CONCLUSIONS
Substantial work on various aspects of PCR testing and microarray detection has accumulated in the literature. EQA programs are crucial if policy makers are to be provided with insight into the level of diagnostic proficiency of responsible laboratories. In addition, due to the multitude of amplification protocols worldwide, reasonable statistical models for evaluation of various protocols are needed (4). Future efforts respecting diagnostic genetic tools should focus on validation, simplified sample treatment, good laboratory practice, establishment of permanent proficiency testing schemes and standardization. Without these additional steps, it will be difficult to implement the data available for routine use. However, in clinical medicine a test result is merely one piece of evidence and should always be interpreted in the light of the clinical assessment. The test may be repeated, another method may be used to confirm the clinical suspicion, or an alternative diagnosis may be pursued.
The work was supported in part by EC grant no. QLK1-CT-1999–00226, grant no. 3401–66–03–5 from the Directorate for Food, Fisheries and AgriBusiness (DFFE), and grant no. 115 from the Nordic Joint Committee for Agricultural Research (NKJ). We should like to thank Stefan Jensen for excellent technical and editorial assistance.
