Abdominal aortic aneurysm repair in New Zealand: a validation of the Australasian Vascular Audit
Abstract
Background
In New Zealand (NZ), there are two major sources of operative data for abdominal aortic aneurysm (AAA) repair: the Australasian Vascular Audit (AVA) and the National Minimum Data Set (NMDS). Since the introduction of the AVA in NZ, there has not been any attempt at the validation of outcome data. The aims of this study were to report the outcomes of AAA repair and validate the AAA data captured by AVA using the NMDS.
Methods
AAA procedures performed in NZ from January 2010 to December 2014 were extracted from the AVA and NMDS. Patients identified from the AVA had their survival status matched to the NMDS. Only primary AAA procedures were included for the analysis, with re-interventions and graft infections excluded. Demographical, risk factors and outcome data were used for validation.
Results
The number of patients undergoing primary AAA procedure from AVA and NMDS was 1713 and 2078, respectively. The AVA inpatient mortality for elective and rupture AAA was 1.6 and 32.2%, respectively. The NMDS 30-day mortality from AAA was 2.5 and 31.5%. Overall, 1604 patients were available for matching, and the NMDS correctly reported 98.1% of endovascular aneurysm repair and 94.2% of elective AAA repairs; however, there were major differences in comorbidity reporting between the data sets.
Conclusion
Both data sets were incomplete, but combining administrative (NMDS) and clinical (AVA) data sets provided a more accurate assessment of mortality figures. More than 80% of AAA repairs are captured by AVA, but further work to improve compliance and comorbidity documentation is required.
Introduction
Clinical governance and accountability requires that operative outcome data is now routinely collected by national health bodies. The majority of surgical units are also required to collect their own data for reporting, audit and research purposes.
Broadly, there are two types of data sources: administrative and clinical. The accuracy and reliability of each is an important issue, and surgeons need to understand the differences between them. The main purpose of each data set differs, and therefore, the variables recorded, the quality and accuracy are likely to differ.1
Surgeons and health policy decision makers rely on end outcomes such as 30-day or 1-year mortality for reporting outcomes. This relatively simple measure can differ depending on the data source. Specifically with cases of abdominal aortic aneurysm (AAA), there is documented variation in 30-day mortality figures for elective repairs depending on the source of the data: prospective population-based reported 8.2% compared to 3.8% from prospective hospital-based.2
The Australasian Vascular Audit (AVA) is a bi-national web-based audit and is the official audit for the Australian and New Zealand Society of Vascular Surgery.3 It collects demographical data, risk factors, operative details and outcomes for all inpatient events on vascular patients. Data entry was commenced in January 2010 with gradual uptake from the majority of vascular units at both private and public hospitals in New Zealand (NZ) and has replaced several individual hospital databases and the Otago Clinical Audit from that date. Since 2012, it has been compulsory for vascular surgery trainees to use AVA to generate their operative logbook.
Understanding the quality and accuracy of data captured by the AVA is important as this audit can provide a useful record of AAA repair. Recently, data quality from Australia submitted to the AVA was subjected to internal validation using a random 5% of major arterial cases, and a reported error rate of 2.6% was found. With regards to external validation, the AVA in Australia captures 63% of the data in the public sector and only 51.6% in the private sector.4
The NZ outcome data submitted to the AVA have not been subjected to any form of validation since its introduction. Furthermore, there are very few published contemporary studies of AAA repair outcomes in NZ; the most recent published information reported a 30-day mortality of 6.7% for elective AAA repaired between 2002 and 2006.5 It is unknown whether this figure included endovascular aneurysm repair (EVAR) or symptomatic but non-rupture AAA. In addition, this figure is considered relatively high compared to contemporary figures, and greater reliance on EVAR in recent practice is likely to have reduced overall operative mortality rates following elective AAA repair.6
Therefore, the aim of this study was to validate the quality and accuracy of demographic and outcome AAA repair data recorded on the AVA using the Ministry of Health National Administrative Data set.
Methods
The Health and Disability Ethics Committee approved this observational study and the obtaining of data from the Ministry of Health National Minimum Data Set (NMDS) for matching purposes. Written individual patient consent was not possible due to the nature of the study design.
Data sources
National Minimum Data Set
In NZ, each patient has a unique seven-digit National Health Index (NHI) that allows matching. A data request enquiry was made for all International Classification of Diseases (ICD)-10 AAA diagnostic codes and procedures from 1 January 2010 to 31 December 2014 (Appendix S1). Patient demographics of up to 20 comorbidity diagnoses and 20 procedures were provided for each hospital encounter. Following this, a data set of unique patients that had a diagnosis of AAA and an AAA-related procedure was developed to represent the number of primary AAA procedures performed.
Australasian Vascular Audit
Between 1 January 2010 and 31 December 2014, all AAA procedures identified from the AVA were obtained. Duplicate patients and secondary procedures were removed, and the primary AAA procedure was considered the index case. The database was checked for procedures performed for graft infections, mycotic aneurysms, isolated iliac aneurysms, EVAR conversions and all re-interventions, and these were excluded from analysis and matching (Fig. S1). Of the NHI identified, all except one was matched with the NMDS database, and three additional fields were returned and added into the AVA data set: ethnicity, deprivation from the 2013 census data and date of death for patients that died and were registered in NZ.
Definitions
Risk factors were defined as outlined in the AVA manual. Inpatient mortality as recorded in the AVA was defined as a death occurring while under the vascular team or if the death occurred in the same hospital admission. The NZ deprivation index is a measure of the level of socioeconomic status (SES) and is measured on a scale from 1 to 10, where 1 is least deprived (better SES) and 10 is the most deprived (worse SES). For the purposes of the validation, it was assumed that demographical and survival status was correctly coded in NMDS, and clinical risk factors and operative details were correctly recorded in the AVA. Hospital volume was categorized into two groups: individual hospitals performing >10% of total AAA repair and individual hospitals performing <10%.
The following information was used for data validation: patient demographics (age and gender), date of admission, length of hospital stay, mode of presentation (acute/arranged) and risk factors (ischaemic heart disease (IHD), diabetes, smoking history and hypertension). For admission dates and dates of birth differences > ±1 day, a manual check across the data sets was performed to ensure that procedures matched.
Statistical analysis
The first 5 years of AVA capture with a minimum 1-year follow-up was chosen as the study duration and follow-up period, respectively. Data validation, cleaning and initial coding was carried out on Microsoft Excel (Microsoft, Redmond, WA, USA), and statistical analyses were performed using SPSS 23 for Mac (SPSS Inc, Chicago, IL, USA). Odds ratio (OR) and 95% confidence intervals (CI) of comorbidities recorded by NMDS compared with AVA data were analysed, and a P-value of <0.05 was considered statistically significant.
Results
During the 5-year period, 1804 AAA procedures were recorded, and following data cleaning and applying the inclusion criteria, 1713 patients were included in the analysis from AVA. There were 6690 hospital encounters in the NMDS, and after removal of duplicates and identifying patients diagnosed with an AAA and had an AAA-related procedure, 2078 patients were included. The overall number of AAA repairs stratified to type of presentation over the years is presented in Figure 1. On average, AVA captured 82.4% of the AAA diagnosis and procedures identified by the NMDS during the 5-year period. Between 2012 and 2014, the AVA capture rate increased to 87.9%. The trend of AAA presentation by method of repair is shown in Figure S2.
, AVA total; 
, MOH total; 
, AVA elective; 
, MOH elective; 
, AVA rupture; 
, MOH rupture.AVA summary
AVA reported 1713 patients who underwent AAA repair: 1220 (71.2%) elective, 207 (12.1%) symptomatic and 286 (16.7%) ruptures. Of the patients who underwent elective repair, 677 (55.5%) had an EVAR. The baseline demographics, comorbidities and operative details of the AVA patients are presented in Table 1. There were no missing risk factors or AAA diameter data as these are mandated fields in AVA data capture.
| Number (%) | |
|---|---|
| Age* (median, range) | 75 (34–93) |
| Males* | 1369 (79.9) |
| Ethnicity | |
| NZ European | 1496 (87.3) |
| Maori | 122 (7.1) |
| Pacific Islanders | 40 (2.3) |
| Other/unknown | 55 (3.2) |
| Deprivation status† | |
| 1–2 (least deprived) | 216 (12.6) |
| 3–4 | 277 (16.2) |
| 5–6 | 402 (23.5) |
| 7–8 | 433 (35.3) |
| 9–10 (most deprived) | 374 (21.8) |
| AAA presentation | |
| Elective | 1220 (71.2) |
| Symptomatic but non-ruptured | 207 (12.1) |
| Rupture | 286 (16.7) |
| IHD | 824 (48.1) |
| Renal impairment | 172 (10.0) |
| Diabetes | 198 (11.6) |
| Hypertension | 1333 (77.8) |
| Smoking status | |
| Never | 535 (31.2) |
| Ex-smoker | 935 (54.6) |
| Current | 243 (14.2) |
| AAA diameter (median, range) | 6.0 cm (3–13.5) |
| EVAR | 778 (45.4) |
| Length of hospital stay (median, IQR) | |
| EVAR | 5 (4–7) |
| OAR | 9 (7–14) |
| Private hospitals | 97 (5.7) |
| High-volume hospitals | 1177 (68.7) |
- * Corrected demographics from NMDS.
- † Missing 11 patients. AAA, abdominal aortic aneurysm; AVA, Australasian Vascular Audit; EVAR, endovascular aneurysm repair; IHD, ischaemic heart disease; IQR, interquartile range; NZ, New Zealand; OAR, open aneurysm repair.
AAA repair outcomes
There were 121 deaths recorded on the AVA. Following matching and verification with the NMDS, an additional six inpatient deaths were discovered but not recorded on AVA: one elective and five ruptured AAA. The overall 30-day mortality obtained from the NMDS was 134/1713 (7.8%): seven patients died between discharge and 30 days after operation (four elective, two symptomatic and one rupture). Mortality stratified for type of repair and presentation is presented in Table 2. The overall 30-day mortality for AVA cases was 2.0% for elective, 5.3% for symptomatic and 34.3% for ruptures. Of the 2078 patients from the NMDS during the study period, the 30-day mortality for non-ruptured and ruptured AAA was 2.5 and 31.5%, respectively.
| Elective 1220 | Symptomatic 207 | Rupture 286 | |
|---|---|---|---|
| OAR | 19/543 (3.5%) | 10/135 (7.4%) | 92/257 (35.8%) |
| EVAR | 6/677 (0.9%) | 1/72 (1.4%) | 6/29 (20.7%) |
| Total (AVA 30 day from NMDS) | 25/1220 (2.0%) | 11/207 (5.3%) | 98/286 (34.3%) |
| IP deaths from AVA (verified from NMDS) | 21/1220 (1.7%) | 9/207 (4.3%) | 97/286 (33.9%) |
| IP deaths recorded on AVA | 20/1220 (1.6%) | 9/207 (4.3%) | 92/286 (32.2%) |
- AVA, Australasian Vascular Audit; EVAR, endovascular aneurysm repair; IP, inpatient; NMDS, National Minimum Data Set; OAR, open aneurysm repair. IP deaths as defined by AVA (discharged from AVA).
Data matching and validation
Of the 1713 patients from the AVA, 1604 matched the hospital episode recorded in the NMDS (93.6%). There were 109 patients that could not be matched, of which 80 patients had an AAA repair at a private hospital, 11 had an AAA repair but were not recorded as an AAA repair or diagnosis in the NMDS, and 18 patients were found on the NMDS, but the AAA encounter did not match the AVA data.
Of the recorded patients who underwent repair in the private sector, 97 patients were recorded in the AVA. There were 57 patients who underwent open aneurysm repair, and 40 patients underwent EVAR with no 30-day mortality observed.
AVA comparison with NMDS data
There were some demographic data errors identified in the AVA. Thirty-nine patients (2.4%) had an incorrect date of birth (error of greater than ±2 days) recorded, and 14 (0.9%) patients had incorrect gender identification. Admission date and length of stay details (error of greater than ±2 days) were incorrect in 33 (2.1%) and 113 (7.0%) patients, respectively.
The NMDS correctly identified 98.1% of the patients as an EVAR and 94.2% as an elective (arranged) admission. Of the comorbidities cross-checked, there was major underreporting in the presence of IHD and hypertension in the NMDS compared to the AVA. The proportion of patients with a smoking history was similar between the two data sets, but there was a 32.8% lack of concordance between them, OR 1.56 (95% CI: 1.34–1.83) P < 0.001. The presence of diabetes, however, was more consistently recorded in both databases (Table 3).
| NMDS (%) | AVA (%) | Discrepancy (%) | Odds ratio* (95% CI) | P-value | |
|---|---|---|---|---|---|
| Males | 1278 (79.7) | 1284 (80.0) | 14 patients | 0.98 (0.82–1.16) | 0.79 |
| Age, mean (SD) | 74.3 (7.8) | 74.2 (8.2) | 39 patients† | — | 0.72‡ |
| Date of admission | — | — | 33 patients§ | — | — |
| Length of stay, median (IQR) | 6 (4–10) | 6 (3.8–10) | 113 patients¶ | — | — |
| IHD | 161 (10.0) | 774 (48.3) | 687 (42.9) | 0.12 (0.10–0.14) | <0.001 |
| Diabetes | 198 (12.3) | 187 (11.7) | 148 (9.2) | 1.07 (0.86–1.32) | 0.55 |
| Hypertension | 563 (35.1) | 1236 (77.1) | 829 (51.7) | 0.16 (0.14–0.19) | <0.001 |
| Smoking history | 1241 (77.4) | 1100 (68.6) | 514 (32.8) | 1.56 (1.34–1.83) | <0.001 |
| Admission type (elective/non-acute) | 1117 (69.6) | 1124 (70.1) | 93 (5.8) | 0.98 (0.84–1.14) | 0.79 |
| EVAR | 745 (46.4) | 737 (45.9) | 30 (1.9) | 1.02 (0.89–1.17) | 0.77 |
- * Odds ratio of data recorded by NMDS compared with AVA data.
- † (> ±2 days), range difference: (−5330 to 31 047) days.
- ‡ T-test.
- § (> ±2 days), range difference: (−173 to 590) days.
- ¶ (> ±2 days), range difference: (−39 to 365) days, excluding 31 non-discharged patients. AVA, Australasian Vascular Audit; CI, confidence interval; EVAR, endovascular aneurysm repair; IHD, ischaemic heart disease; IQR, interquartile range; NMDS, National Minimum Data Set.
Discussion
In this study, by interrogating both the national data set and the vascular surgery audit, we were able to provide accurate 30-day outcomes, describe the national burden of AAA on health services and present sources of error. Both data sets were incomplete but the early mortality was similar. Regulatory bodies are very likely to use the most accessible data rather than the ‘best’ available data when policy decisions are made; therefore, understanding the limitations of each data set is important.
In NZ, the viability of screening for AAA is being investigated, and being able to provide an accurate estimate of local figures is important. The change of AAA epidemiology and the relatively lower early mortality achieved nowadays can alter the clinical decision making for AAA management in comparison to evidence-based norms established two decades ago.7
The mortality rates for AAA repairs from both data sets were very similar, and the results compare favourably with reported contemporary international elective AAA repair figures.6 However, differences between the two data sets might be attributable to the unclassified diagnosis of ‘symptomatic’ but non-rupture AAA, which occurred in about 12% of AAA presentations. The majority of private AAA procedures performed were not found on the NMDS. Therefore, the total number of repairs performed in the private sector is unknown. Excluding private AAA repairs from national figures could also partially account for the 0.5% increase in mortality reported in the NMDS.
This type of validation study has been assessed in other studies in different geographical settings. Several similar studies linking administrative and clinical databases have been conducted in the UK, and conflicting results have been reported. Holt et al. compared 1102 elective AAA patients from the English Hospital Episode statistics with clinical case records, and 86% of the cases were confirmed as an elective AAA repair.8 Johal et al. reported that of patients undergoing AAA replacement, the diagnosis of AAA was consistent in >90%.9 However, a study from Scotland highlighted that such a discrepancy between clinical and national data can lead to a significant reduction in reported mortality from national figures.10 Our results were similar to these studies, but including both data sets allowed us to present more accurate outcome data.
As with the majority of databases, the quality and accuracy of data depend on those entering data. There has been a gradual uptake of AVA usage in some NZ vascular units and among surgeons; hence, the low capture in the first 2 years can be expected. In Australia, the AVA compliance dropped by about 10% in 2013.4 In NZ, data compliance in 2014 decreased without a notable systematic reason. Continued auditing of the AVA data requires future monitoring and improvement.
With regards to the comorbidities comparison, where there was a wide range of definitions, large differences were observed (such as in hypertension and IHD), resulting in high discrepancy rates, whereas with comorbidities with a clearer definition (such as the presence of diabetes), the differences were smaller. The consistency of diabetes coding has been also been shown in a similar study.8 This might be due to the coding nature of diabetes that coders are well trained to enter from case and discharge notes.
Strengths and limitations
The relatively large patient number during the study period to represent the vascular surgeon workload for AAA disease is of high importance to governing bodies. For this reason, we decided to include all AAA repairs (complex and standard) to provide realistic estimates of mortality and increase the generalizability of these figures that would more accurately reflect contemporary clinical practice.
The AVA webpage form is currently not linked to hospital coding software, and therefore, human typing errors are very likely to occur, particularly in the demographics section of the free typing, that is, date of birth and gender. These errors could potentially be avoided by amalgamating existing software platforms. However, with numerous different software used in each institution it is unlikely that this will occur.
Some important prognostic information such as AAA diameter and renal impairment could not be verified as these variables are not recorded on the NMDS. In addition, the timing of data entry into the AVA with respect to the admission details was not available. Further work to determine if such delays in data entry might lead to a source of error is of merit.
Information on aspirin and statin use is not collected in the AVA; given the importance of risk factor modification, inclusion of such variables would be useful in auditing and model development. Other recorded data such as hypertension is not considered to be a significant risk factor per se, but rather whether it is treated or not. Hence, routine collection of ‘hypertension’ is perhaps no longer an important comorbidity for mortality prediction. Respiratory disease and cardiac failure have greater impact on short- and long-term outcomes, and inclusion into the data requirement might better inform risk models and decision making.11
Conclusions
Both AAA data sets were incomplete, but this analysis has allowed us to understand the differences. The AVA captures more than 82% of AAA repairs performed in NZ. Matching clinical databases and national administrative data sets provides a better representation of absolute national workload, provides accurate survival status and increases the utility of a single data set to reflect real-world outcomes.
Acknowledgements
Manar Khashram is a PhD candidate supported by the Foundation for Surgery New Zealand Research Scholarship. The authors acknowledge the Analytical Services at the Ministry of Health, New Zealand for providing the data extracts used for analysis in this study and Mr Ziad Khashram for data preparation and presentation. We thank all the vascular surgeons and vascular trainees who have contributed their time and effort in submitting data into the AVA particularly the following vascular surgeons: A. Hill, S. Caldwell, P. Haggart and R. Evans for assisting with data retrieval.
