The presence of a cytopathologist increases the diagnostic accuracy of endoscopic ultrasound-guided fine needle aspiration cytology for pancreatic adenocarcinoma: a meta-analysis

Literature search

A systematic literature search was performed to identify studies assessing the diagnostic value of EUS-FNA for pancreatic ductal carcinoma. The MEDLINE and SCOPUS databases, from January 1994 to March 2011, were searched with the following keywords: (‘EUS’ OR ‘endoscopic ultrasound’ OR ‘endoscopies’ OR ‘ultrasonic endoscopy(ies)’) AND (‘pancreas OR pancreatic’ OR ‘adenocarcinoma OR pancreatic adenocarcinoma’ OR ‘ductal carcinoma OR pancreatic ductal carcinoma OR ductal carcinoma of the pancreas’) OR (‘solid tumour(s)’ OR ‘solid mass’ OR ‘solid lesion’) AND (‘accuracy’ OR ‘sensitivity OR specificity’ OR ‘true-positive OR true-negative OR false-positive OR false-negative’).

Other databases, such as Embase and the Cochrane library were also checked for relevant articles with similar key terminology.

Study selection

Two investigators independently reviewed all returned-search literature (S.H. and I.E.). Accordingly, all abstracts were read to select for potentially eligible articles. Subsequent retrieval of the full text of these articles was performed to further determine eligibility. Non-English literature returns were excluded. The inclusion and exclusion criteria are shown in Table 1. According to our strict criteria, studies with potentially overlapping study populations were excluded. The year of publication was very useful as we included only the study with the largest patient population and published latest in time, whereas previous studies by the same author, with often a smaller subgroup of patients, were excluded unless we could determine the studies were independent.

The study designs (retrospective/prospective) were determined to be acceptable or unacceptable if the patients were or were not consecutively included, whether or not the test results were blindly interpreted, and dependent upon the type of reference standard that was used in the study.

Data collection and extraction

From the selected studies that met the strict inclusion criteria, two independent investigators extracted the following data onto standardized data forms utilizing Microsoft Excel: design, author, year of publication, sample size, clinical context, the selected study's inclusion and exclusion criteria, reference standard, number of passes, presence of cytopathology assessment and the number of true-positive (TP), true-negative (TN), false-positive (FP) and false-negative diagnoses (FN). 2 × 2 tables were constructed using the numbers of TP (a_i), FP (b_i), FN (c_i) and TN (d_i) diagnoses in the detection of pancreatic ductal carcinoma.

Quality assessment

Quality assessment of diagnostic accuracy studies (QUADAS), an evidence-based quality assessment tool used in systematic reviews of studies of diagnostic accuracy, was used to evaluate the risk of bias and variability of the studies and extract relevant study characteristics.3 The Cochrane handbook for assessing diagnostic accuracy in systematic reviews suggests the use of 11 out of 14 QUADAS items.3 The following eleven QUADAS methodological quality questions were assessed: (1) Was the patient spectrum representative of an actual patient population in clinical practice? (2) Was the reference standard acceptable? (3) Was the period of time between the index test and the reference standard limited, so that it was less likely that the target condition changed/evolved between the two tests? (4) Were all patients or just a subset (partial segment) evaluated with the reference standard? (5) Did all patients receive the same reference standard, regardless of the index test results? (6) Were the results from the index test interpreted without knowledge of the reference standard results? (7) Were the index test results not incorporated into the reference standard? (8) Were the reference standard results interpreted while blinded to the index test results? (9) Were clinical data analogous to what would be available in practice presented? (10) Were uninterpretable results mentioned? (11) Were withdrawals or exclusions explained?

Statistical analysis

Computational analysis of the data was performed using Cochrane Diagnostic Accuracy protocol with Review Manager (RevMan5, Cochrane Diagnostic Accuracy Group, Birmingham, UK), Mix 2.0 (Sunnyvale, CA, USA), and MetaDiSc (Madrid, Spain) to analyse the data for EUS-FNA of pancreatic ductal carcinoma. Utilizing the values from the 2 × 2 tables, the sensitivities and specificities were calculated and pooled as follows:

The likelihood ratio for a positive test result (LR+) was calculated by dividing the sensitivity by the false-positive error rate, whereas, the likelihood ratio for a negative test result (LR−) was calculated by dividing the false-negative error rate by the specificity. The diagnostic odds ratio (DOR) was calculated by dividing the (LR+) by (LR−). In addition, summary receiver-operating curves (SROC), where sensitivity is the y-axis and 1-specificity is the x-axis, were produced. The closeness of the area under the curve (AUC) to 1.0 is a validated depiction of the diagnostic accuracy. The variation in accuracy measures in the individual studies and in the pooled measures were displayed graphically using forest plots which visually show the amount of variation between the sensitivity and specificity of the individual studies, as well as an estimate of the overall accuracy of all the studies combined.

Heterogeneity was analysed by performing subgroup analysis, comparing different characteristics among the included studies. Studies were evaluated by geographic location (continent of origin), study design (retrospective or prospective), sample size, presence of on-site cytopathologist, reference standard and QUADAS score. QUADAS scores were calculated as follows. Each study was assessed for the specified QUADAS components; a score of +2, +1, or 0 was assigned for each QUADAS criterion that was met, undetermined, or not met, respectively. An additive sum of the assigned points determined a final score for each study. A funnel plot of standard error was created to assess for publication bias. ORs of the included studies were plotted according to the variance of the log OR estimate. The x-axis consisted of the natural logarithm of the diagnostic OR, whereas on the y-axis, we plotted the sample size. Smaller studies (with fewer patients) that have less precision in estimating the underlying OR will scatter widely, with a narrowing among larger studies. In the absence of publication bias, the plot resembles a symmetrical funnel. If there is bias however, the funnel plot will be asymmetrical.

Literature search and selection of studies

Our initial search yielded 683 study titles and abstracts (465 MEDLINE, 218 SCOPUS). Of these, 34 studies met the pre-defined inclusion and exclusion criteria and were selected for meta-analysis. Six hundred and forty-nine were excluded because (1) the articles were duplicates from the other database or non-English publications (n = 412); (2) the aim of the study was not to reveal the diagnostic value of EUS-FNA for identification and characterization of pancreatic carcinoma, particularly ductal adenocarcinoma (n = 180); (3) insufficient data were reported to construct a 2 × 2 table (n = 44); (4) researchers in the articles did not use a proper reference standard or follow-up interval of equal to or greater than 5 months (n = 8); (5) the full-text article could not be obtained because the journal was no longer in publication (n = 3); and (6) statistical analysis could not be performed with the data provided (n = 2). Selection of our studies is illustrated in Figure 1. The full text of all 34 studies was read for complete examination of their contents.4-37

The 34 studies included 3644 patients, with 2285 pancreatic adenocarcinomas (Tables 2 and 3). Seventeen studies were noted to be retrospective and 17 prospective. Of the 34 selected, 29 used EUS-FNA alone, two used combined EUS-FNA and EUS TruCut, and three combined EUS-FNA and CT FNA, EUS-FNA, CT FNA and US FNA, and EUS-FNA and contrast harmonic echo (CHE-EUS), respectively. In all studies, the cytopathologists were blinded to the diagnostic findings from previous or concurrent studies and information was limited to clinical presentation and radiographic studies alone.

The reference standard, in the majority, was clinical and/or histological findings and only three studies used histology alone. On-site evaluation by the cytopathologist was present in 16 studies, sometimes in five, was not indicated in three and not present in ten. The time of follow-up for all the 34 studies was longer than 9 months on average, unless diagnosis was confirmed by histology, the patient was lost to follow-up, or expired prior to the stipulated period.

Methodological quality assessment

All the selected studies were of moderate to good quality based upon meeting seven or greater scoring criteria of the modified QUADAS 11 question based assessment. The mean QUADAS score was 18.2 with a range of 15–21 based upon our scoring system. A representative spectrum of patients was thought to be included in all studies as all studies reported a consecutive patient population. Histology was used as the reference standard in three studies (8.8%); partial verification bias (i.e. a portion of the study group not subjected to the defined reference standard) was present in two studies (5.9%), where positive (malignant) cytology was considered diagnostic and included in the reference standard. The risk of differential verification bias (i.e. verification using different reference standards) could not be avoided in 29 studies (85.3%), which used varying reference standards including clinical follow-up and/or histopathology. One study used repeat EUS-FNA as the reference standard causing incorporation bias. The existence of uninterpretable or intermediate results and withdrawals, if present, were explained in all studies. In addition, none of the studies clearly stated the interval time between the index test (EUS-FNA) and diagnostic confirmation. Publication bias (i.e. the selection of articles) did not impact the diagnostic accuracy of this meta-analysis, as shown by a relatively symmetrical funnel plot (Figure 2).

Estimates of diagnostic accuracy

The sensitivity of EUS-FNA for the diagnosis of pancreatic adenocarcinoma ranged from 0.50 to 1.00 and the specificity ranged from 0.88 to 1.00. The pooled sensitivity for EUS-FNA of the pancreas for pancreatic adenocarcinoma was 88.6% [95% confidence interval (CI): 87.2–89.9] and the pooled specificity was 99.3% (95% CI: 98.7–99.7). The pooled LR+ was 33.5 (95% CI: 20.76–53.91) and the pooled LR− was 0.11 (95% CI: 0.08–0.16). The DOR expresses how much greater the odds of having the disease are for the people with a positive test result than for the people with a negative test result. The pooled DOR was 383.64 (95% CI: 219.66–684.09) showing a very high accuracy of EUS-FNA in diagnosing pancreatic ductal carcinoma. The (SROC) curves for overall accuracy and the AUC are shown in Figure 3.

We evaluated heterogeneity by plotting the sensitivity and specificity from each study on a forest plot and noted how well the CIs overlapped. Sensitivity was notably lower in most studies that lacked on-site cytopathology evaluation, whereas only two of 14 studies with a sensitivity of 95% or higher did not have on-site cytopathology. The forest plot for the sensitivity EUS-FNA is shown in Figure 4. There were only nine false-positive diagnoses of which six came from a single institutional study; cytopathology was sometimes present for that particular study. Owing to either only one or no false positives in the other studies, a forest plot of specificity is not shown. Although the studies in which rapid on-site evaluation (ROSE) was performed tended to have a lower inadequate (unsatisfactory) rate (mean = 0.03, 95% CI 0.01–1.96), this difference does not reach statistical significance compared with studies without ROSE (mean = 0.05, 95% CI 0.01–1.96). Thirteen studies did not report any inadequate specimens. Inadequate specimens were included in the assessment of sensitivity and specificity in the assessment of diagnostic performance in four studies.

Univariate analysis was performed on the different variables considered as possible sources of heterogeneity; the presence of a cytopathologist for ROSE (P = 0.012), studies with over 50 patients (P = 0.013) and the type of reference standard used (P = 0.015) were found to be statistically significant (Table 4). Factors that were not found to have a significant contribution to performance variability included: location, QUADAS score and study design. SROC for the presence of cytopathology is shown in Figure 5. Meta-regression was performed on these three variables (presence of on-site cytopathology, reference standard and size) and only the presence of on-site cytopathology (P = 0.001) was shown to be independently significant after correction (Table 5).

Applicability of findings to clinical practice

EUS-FNA has been shown to detect tumours less than 3 mm, which are not readily recognized by imaging. Therefore, the results of this study confirm the likelihood through a systematic review of over two thousand cases of pancreatic adenocarcinoma diagnosed by EUS-FNA at various institutions is principally accurate. This is especially of concern when healthcare management and healthcare cost initiatives desire less invasive, cost efficient modalities, which would not diminish the current standard of practice.

Equally important is the fact that this analysis shows that even although the diagnostic accuracy of EUS-FNA is high, it is not as high as previously purported when using histology the ‘true’ gold standard. This may be the result of overconfidence in the observer, imaging analysis and limited clinical follow-up. It is not unforeseeable that a poorly differentiated tumour could have been called pancreatic adenocarcinoma by EUS-FNA and the patient subsequently dies shortly afterwards with this as the causative agent. However, hypothetically, if procurement of tissue showed poorly differentiated neuroendocrine carcinoma or lymphoma the diagnostic value would change. Moreover, if the lesion was called pancreatic adenocarcinoma but was truly pancreatitis, yet the patient die 3 months later from unrelated causes this could erroneously be misinterpreted in clinical follow-up.

We believe the most important point shown by this meta-analysis is the presence of on-site cytopathology assessment yields higher diagnostic accuracy. We also think this meta-analysis shows that larger size centres and reference standard use are not significant but the accuracy lies in the skill of the endosonographer. Our own experience has shown that many of the referrals we received were secondary to the inexperience of the centre where the initial procedure was performed and the presence of the cytopathologist in the room at the time of aspiration prevents inadequate samples and reduces the likelihood of sampling error.