Accuracy of quantitative HIV-1 RNA test methods at 1000 copies/mL and the potential impact of differences in assay calibration on therapy monitoring of patients

1 INTRODUCTION

In the last decades, a World Health Organization (WHO) advisory group on HIV infection recommended guidelines for patient management based on viral load (VL) monitoring during antiretroviral therapy (ART). The WHO guideline recommends VL greater than 1000 copies (cp)/mL to be detected in two sequential samples as an indication for lack of viral control by ART, although lower concentrations might already be considered indicative of virologic failure.¹ This threshold value was chosen in consideration of the wide utilization of dry blood spots (DBS) as preferred sample format for VL measurement in sub-Saharan Africa (SSA). DBS facilitates sample transport to central laboratories but reduces sensitivity and accuracy of VL measurement in the range around 1000 cp/mL.^2-6 In some SSA countries, the threshold value was decreased to 400cp/mL, a level only feasible for VL detection on fresh or frozen stored plasma samples.⁷ However, in most SSA countries, the clinicians base their therapeutic decisions on whether or not the HIV-1 VL result is above or below 1000cp/mL according to the WHO guideline.¹ It is therefore important to determine the accuracy of multiple quantitative HIV-1 RNA assays at this clinically critical level to assess the potential risk of misclassification of ART outcome.

To conduct such an assessment, a well-calibrated HIV-1 subtype B standard established in the 1990s (Viral Quality Control [VQC]; Sanquin, Amsterdam, The Netherlands) was used for preparing 1000cp/mL aliquots (ViraQ HIV-1 Quant 1000; BioQControl, The Netherlands). The VQC standard was calibrated in nucleic acid copies in multiple VL assays as well as in IUs against the first and second International Standards in two WHO collaborative studies.^{8, 9} Since 1995 the VQC standard was widely used in nucleic acid testing (NAT) proficiency and validation studies (mainly for estimating the analytical sensitivity of blood screening assays and estimation of the infectious window period of NAT-screened blood donations).^10-14

In 2018 VQC reference samples of 1000cp/mL were distributed to five laboratories using different commercial VL assay systems. Each laboratory tested 24 samples in four test runs of six aliquots and the HIV-1 VL results on the five assays were compared. The quantitative results were also compared with those obtained on the same VQC HIV-1 subtype B standard by the quantitative assays that were in use in the late 1990s and early 2000s (at the time of introduction of the first and second WHO International Standards). Since then, even without the use of DBS, the variation in VL results reported by the present assays was larger than the expected; we tried to understand the observed differences by reviewing the calibration data in the package inserts of the HIV-1 RNA assays. Some of the assays were calibrated in IUs against the first and second native HIV-1 International Standards and others against the third WHO standard (which like the subsequent fourth WHO standard has been heat-treated before lyophilization^{15, 16}). All manufacturers reported HIV-1 VL in cp/mL and give copy/IU conversion factors in their package insert. Some assays had been calibrated in copies using another widely used standard (VQA, NIH).¹⁷ The frequent replacement of WHO standards over the last two decades made a certain drift in the amount of HIV-1 RNA per IU inevitable; this is because of the uncertainty in calibration in the WHO collaborative studies as well as by changing the source material for preparation of the standards (introducing nucleic acid sequence differences or matrix effects by heat treatment and lyophilisation). Therefore, we also examined the impact of heat inactivation of the VQC standard in different assays and compared the data with those reported on the untreated and later heat-inactivated WHO standard batches. Finally, as the commutability of HIV-1 standards to regular clinical samples has not been well examined, we also compared the relative quantification of the native VQC and second WHO standards with the Bland-Altman agreement values reported in large number of patient samples in different VL method comparison studies.^18-27 Our study underlines the limitations in calibration of VL assays based on the current system of biological standardization and discusses its impact on the interpretation of a recommended 1000cp/mL threshold value for clinical evaluation of ART efficacy.

2 MATERIALS AND METHODS

3 RESULTS

3.1 Quantification of 1000cp/mL reference samples by five HIV-1 RNA assays

Figure 1 shows the distribution of quantitative results of five VL assays on 1000cp/mL VQC reference samples in four runs of six tests. The 24 test results of four of the five assays clustered within the expected range of 1000 ± 0.3log (500-2000) cp/mL except for one outlier value of 4617cp/mL were obtained with the Abbott RealTime Assay. By contrast all but one of the 23 available test results obtained with the Cepheid GeneXpert assay were found above this range. When the geometric mean values (and 95% CI) of all 24 VL tests were calculated (Table 3) it was found that the Cepheid assay reported 2.5 (1.7-3.7)-fold higher values than those assigned to the VQC standard, whereas the Abbott RealTime Assay reported VL results that were closest to the nominal value, that is, 1084 (784-1572) cp/mL (after the one outlier result had been excluded from the data set). The other assays of BioMerieux, Roche, and Hologic reported 1.1 (0.5-2.2), 1.3 (0.9-1.8), and 1.6 (1.3-2.0)-fold higher values than the nominal value, respectively. The Hologic Aptima assay was the most precise showing a 95% CI of 82%-122%, followed by Abbott, Roche, Cepheid, and BioMerieux with 95% CIs of 72%-163%, 70%-143%, 68%-146%, and 50%-201%, respectively.

Table 3. Geometric mean values and 95% confidence interval (CI) calculated for 24 replicate test results obtained in 2018 by five HIV-1 viral load assays in 4 test runs (individual data shown in Figure 1)

HIV-1 VL assay	n	Geomean, cp/mL	95% CI,a cp/mL	95% CI,a %
Abbott RealTime	24	1084b	784-1572b	72-163b
Hologic Aptima	24	1616	1324-1973	82-122
Roche CAP/CTM	24	1277	892-1828	70-143
Cepheid Gene Xpert	23	2502	1713-3655	68-146
BioMerieux NucliSens EasyQ	24	1110	533-2230	50-201

• ^a T factor of 2.07 used for calculating 95% confidence bounds using average and SD on log viral load values.
• ^b One outlier result of 4617 cp/mL was excluded from the calculations.

4 DISCUSSION

In SSA countries, the current VL assays are usually applied in combination with DBS,^2-6 which increases the 95% LOD from below 50cp/mL to near the 1000cp/mL threshold value recommended by the WHO for making therapeutic decisions in ART monitoring.¹ This comparison study indicates that even with direct testing of plasma samples there is substantial variation in quantitative results reported by five widely used VL assays at the 1000cp/mL threshold value. We found up to 50%-200% confidence bounds in assay precision and up to 2.5-fold differences in assay calibration when assessed against the VQC subtype B standard. The geometric mean values (95% CI) reported on 24 replicate tests on the 1000cp/mL run control samples varied from 1084 (784-1572) in the Abbott RealTime assay to 2502 (1713-3655) in the Cepheid Xpert assay. Similar variation in VL results was found in the clinical evaluation studies comparing different VL assays in testing subtype B patient samples, although differences in VL were usually greater with subtype C and other non-B subtypes.^18-27

The Hologic Aptima reported 1.5 (1.2-1.9)-fold higher values than the Abbott RealTime assay on the VQC standard, but when the second WHO 97/650 standard was used this difference reduced to 1.1 ((0.8-1.3)-fold. By contrast the Roche CAP/CTM quantified both the VQC and second WHO standard with 1.3-1.4-fold higher values than the Abbott assay. However, the quantification of the heat-inactivated version of the VQC standard was significantly (1.9- to 2.3-fold) higher by the Hologic and Roche assays than by the Abbott assay. Interestingly, the WHO collaborative study report¹⁶ also showed 1.9- to 2.2-fold higher copy/mL values with the heat-treated third WHO standard in the Hologic Aptima and Roche CAP/CTM v2.0 assays than in the Abbott RealTime assay. Hence, pasteurization of HIV-1 in VQC and WHO standards had a destandardizing effect. This can be explained by differences in efficiency of nucleic acid extraction methods between assays and by aggregation of viral particles and proteins in the heat-treated standard. In addition, lyophilization of HIV-1 in plasma may affect virion integrity and the natural configuration of the virus in EDTA plasma as we previously observed that VL recovery after lyophilization (and rehydration) was even lower than after pasteurization (28% vs 56%).²⁹ The latest third and fourth WHO standards are both heat-treated and lyophilized,^{15, 16} and this likely has an impact on the relative quantification with different VL assays and the copy/IU conversion ratios reported in the manufacturers' package inserts. We suspect that the higher copy number reported using the Cepheid Xpert and Hologic Aptima assays is related to calibration against the third heat-treated WHO standard instead of the untreated first and second WHO standards that was used for calibration of the Roche cobas CAP/CTM, Abbott RealTime, and BioMerieux NucliSens assays. Unfortunately, the calibration system of WHO replacement standards does not follow the ISO 17511 guideline for metrological traceability of measurement values in biological samples and does not report uncertainties in calibration of replacement standards in arbitrary IUs.³⁰ The continuity of the amount of HIV-1 RNA per IU cannot be guaranteed as the calibration of a newer standard batch in the WHO collaborative studies does not include the first established standard but only the latest replacement standard. This may be the reason why NAT manufacturers, so far, do not report quantitative results of HIV-1 VL assays in IUs but still calibrate their assays in nucleic acid copy numbers based on their own standards. In one cross-calibration experiment, we observed a significant drift in cp/IU of 19% with the third and 39% with the fourth WHO replacement standards as compared with the second WHO standard in the Abbott RealTime assay. A similar drift in copy/IU ratios as in our calibration experiment was also found in this as well as in other VL assays when reviewing the quantitative data on the third and fourth International Standard in the latest WHO collaborative study report showing 11%-37% lower copy/IU values by different VL assays on the replacement standard. Hence, the system used by WHO for biological standardization with different batches of lyophilized reference ampoules over time is not ideal for consistent calibration of NAT methods. Therefore, we believe that a liquid HIV-1 plasma standard stored at −80°C and available in sufficient supply for many decades (such as the VQC and VQA standards) are instrumental in providing a second anchor in assay calibration.

The data in this study demonstrate that there is not only considerable variation in quantitative values reported by the assays on one standard but also in the relative quantification of different standards (even for standards of the same subtype, ie, HIV-1 subtype B on which all assays are standardized). It is not known to what extent the different viral standards examined in this report are commutable to natural HIV-1 subtype B samples and therefore we compared relative quantification of VQC and WHO standards by different assays with Bland-Altman agreement factors observed in a number of clinical studies.^18-27 These studies demonstrated up to twofold differences in relative quantification of subtype B samples and more than threefold on some of the non-B subtypes. To our knowledge there are no data that compare the relative quantification of 1000cp/mL reference samples of different subtypes and circulating recombinant forms (CRFs), but Manak et al²² compared three assays on clinical (and cultured) HIV-1 samples of different subtypes and showed significant differences in relative quantification depending on the subtype. A 20-member panel of different subtypes and CRFs (P0140 HIV 1000cp/mL subtype reference panel, BioQControl) is available and has been used by some manufacturers for performance evaluation without publishing the data. The limited available data on this multiple subtype reference panel show significant differences in relative quantification of subtypes (unpublished results). So far, the Common Technical Specifications of the EU/IVD Directive/Regulation do not set requirements for analytical sensitivity or quantification of NAT methods for different genotypes. The data presented in this report illustrate that the quantification on one WHO standard batch is not predictive for the performance on other standards and clinical samples, even of the same subtype. Clearly, there are fundamental limitations in using only one standard for assay calibration and the WHO standardization system becomes even weaker when international standards are frequently replaced without being able to relate to a higher order standard with a known number of virions (and double amount of nucleic acid copies as HIV is a diploid virus). The VQC and VQA standards have been calibrated 25 years ago in nucleic acid copies and the VQC standard was also calibrated in IUs assigned to the first and second International Standards in the first WHO collaborative study.⁸ These liquid viral standards in plasma cannot be considered higher order standards but are instrumental for assay calibration based on the copy numbers historically assigned to these standards. The VQC standard is still available for future WHO calibration studies, which—if properly designed—could help minimizing quantification bias between VL assays as much as possible.

What are the implications of these considerations for the interpretation of quantitative results of HIV-1 RNA assays when classifying patients with VL above or below the 1000cp/mL threshold recommended by WHO in therapy monitoring? This study shows that there are several limitations in assay standardization, reducing the accuracy of VL assays in reporting copy numbers up to 2.5-fold, whereas on top of that assay precision (95% CIs) is limited to 50%-200%. We, therefore, propose that clinicians use an inaccuracy range from 500-2000cp/mL rather than a distinct 1000cp/mL cutoff value to identify patients with possible ART breakthrough infection or with poor adherence to drug-taking schemes. Lack of viral control may be confirmed when VL has increased to a level between 500cp/mL to 2000cp/mL. It is proposed that further studies are performed to compare VL assays with and without DBS on 1000cp/mL reference samples of multiple subtypes and CRFs to obtain an even better understanding of how the WHO recommended 1000cp/mL alert level in therapy monitoring should be interpreted by clinicians. If certain assays show deviating results from the others, on multiple subtypes, recalibration should be considered by the manufacturer to harmonize VL detection worldwide.

CONFLICT OF INTERESTS

Authors are owners of a company (BioQControl) that manufactures run controls and reference panels for performance evaluation of infectious disease tests.