Assessing heterogeneity of electronic health-care databases: A case study of background incidence rates of venous thromboembolism

1 INTRODUCTION

In the past two decades, the usage of large health-care databases has increased greatly.¹ Regulatory agencies such as the European Medicines Agency (EMA) and the US Food and Drug Administration (FDA) have highlighted the value of real-world data (RWD) in medicines regulation.^{2, 3} In Europe, the initiation of the Data Analysis and Real World Interrogation Network (DARWIN EU),⁴ as well as the European Health Data Space are changing the landscape of real-world evidence (RWE) generation towards multi-database studies.

While there are already a number of advantages in using RWD in regulatory decision-making, those benefits can be improved by using more than one data source.⁵ Trivially, incorporating data from multiple data sources in an analysis will increase sample size. This can be crucial in situations with low event counts, such as for estimating the incidence rates (IRs) of a rare disease. While observational data are more generalizable to the real world than randomised controlled trials, the level of generalizability can be even further increased, by covering a broader and more representative population, thus possibly mitigating selection biases that are specific to single databases and by allowing for the quantification of true differences between populations.

Even though the benefits of using multiple data sources cannot be denied, data from those sources should not be pooled without a preliminary assessment of the suitability of pooling data, due to inherent differences in their characteristics. Simulations have shown increased risks of false-positive and false-negative safety signals when pooling data from multiple databases.⁶ This heterogeneity can have multiple forms, some of which are desirable for understanding true differences in outcomes event rates, while others make interpretation of results and decision-making regarding selection of suitable background rates for specific purposes such as observed-to-expected analyses for vaccines highly challenging.⁷

Sources of heterogeneity can be categorized into three types: measurement heterogeneity, information heterogeneity (both may be considered methodological heterogeneity), and true heterogeneity (also called clinical heterogeneity).⁸ While measurement (e.g., clinical classification systems) and information heterogeneity (e.g., granularity of clinical codes) can generally be considered undesirable, clinical heterogeneity has its value, for example, by improving external validity of results or understanding differences in prescription patterns or impact of risk minimisation measures (RMMs) between different geographical regions, health-care systems or behaviours. However, to understand heterogeneity, it is important to use appropriate tools to detect, report and account for it.

During the COVID-19 pandemic, RWD rapidly provided impactful evidence on safety and effectiveness of therapeutics and vaccines.⁹ This included the generation of background IR for adverse events of special interest (AESIs) for COVID-19 vaccines.¹⁰ Those background rates continue to be used in observed-to-expected analyses to estimate the expected number of cases in the general population prior introduction of COVID-19 vaccination, or during SARS-CoV2 circulation in non-vaccinated populations.

The list of AESIs included the concepts of deep vein thrombosis (DVT) and pulmonary embolism (PE). These two concepts make up the term venous thromboembolism (VTE).¹¹ EMA pharmacovigilance activities identified VTE as a possible adverse event of Jcovden (former COVID-19 Vaccine Janssen)¹² and listed VTE as an adverse event of Vaxzevria.¹³ Background rates were used to calculate the excess number of cases potentially linked with these vaccines. However, the reported background rates showed large differences between EU countries as reflected through national health-care records.^{14, 15}

The objective of this study was to explore data characteristics that trigger heterogeneity in IR through both descriptive and statistical measures, using VTE as a case study. This investigation will further provide support in selecting adequate statistical methods for handling heterogeneity when pooling observations to derive meaningful pooled estimates to support regulatory decision-making.

2 METHODS

3 RESULTS

A total of 60 080 169 subjects contributed to the 13 databases. See Tables A2 and A3 for an overview of the reported IRs stratified by age category and gender, by database and by study.

Two databases (CPRD GOLD and SIDIAP) were used in both studies by both consortia. Differences in defining the study cohort resulted in the cohort entry criteria not being identical. After removing the duplicated databases, a total of 54 257 284 subjects derived from 11 databases were included in the main analysis, representing collectively all age and gender subgroups from six countries.

As the first step, we display the age–gender-database-specific IR estimates in a forest plot (Figure 1) using total population estimates. The forest plot showed a relatively large amount of heterogeneity between databases and within strata of age groups and gender. While for the 0–19 age group the IRs appeared to be in the same order of magnitude, with increasing age and increasing IRs the heterogeneity increased. In the 80+ age group, estimates ranged from <100 to >1 000 per 100 000 person-years. There did not seem to be a large difference in heterogeneity between gender categories. Two databases, LPD France and PHARMO, showed considerably lower estimates than the other databases in most age groups. The Dutch IPCI database, on the other hand, showed the highest estimates, especially for younger age groups and women. We generally observed consistent ranking of IR estimates across age groups, that is, across strata estimates are consistently high or low relative to the other databases.

Next to the age–gender-database-specific IRs, age–gender IRs from meta-analyses were calculated. In our study, the meta-analysis estimated IR of VTE from 5 to 421 per 100 000 person-years depending on age–gender strata. The wide confidence interval of the summary (i.e., pooled) measure identified even within each stratum large patient-level differences. Table A4 in the appendix displays the age–gender IR estimates and CIs of the pooled measure for VTE from meta-analyses.

Figure A1 in the appendix shows the calculated prediction interval for the primary analysis. The prediction intervals for each age–gender group were notably high confirming the substantial population-level heterogeneity observed across data sources.

Table 2 shows the estimated I² and τ² values by age–gender stratum. The values for I² indicated that a majority of the observed variability is due to differences between databases rather than random sampling error. Supporting the impression from the forest plot, we observed an increasing estimate of I² with increasing age in both sexes. There did not seem to be an age-related trend in the estimates of τ², but estimates for τ² appear to be lower for males than for females.

3.1 Sensitivity analyses

In the first sensitivity analysis, we restricted the databases to those with PC-H linkage (Figure 2). The forest plot demonstrates a relatively large decrease in heterogeneity when restricting the analysis to databases with PC-H linkage, across all age groups. It became apparent that this restriction of databases primarily leads to low estimates being excluded from the analysis. In Table A5 in the appendix, which lists I² and τ² estimates for all sensitivity analyses, we noticed lowered I² estimates especially for younger age groups and considerably lowered τ² values for all age groups. Figure 3 did not imply any reduction in heterogeneity when restricting the analysis to databases using ICD 10 codes to diagnose VTE. Both range and distribution of estimates were similar to the primary analysis. The same was true for estimates of I² and τ². Figure 4a, b showed forest plots of the analysis considering data from ACCESS and ERASMUS separately. Note that due to including the estimates from both the total population and the sub-population with PC-H linkage, there was some dependence between the estimates of BIFAP, SIDIAP, and PHARMO. For ACCESS, the hospital database PHARMO showed far lower estimates than all other databases included. This could be linked to an oversampling of the denominator. Apart from this, visually there seemed to be some reduction in heterogeneity. Data from ERASMUS and ACCESS showed a similar amount of heterogeneity; the forest plots indicate that estimates from ERASMUS are spread more equally, while the PHARMO hospital data differ strongly from the other estimates within ACCESS.

4 DISCUSSION

This study explored heterogeneity in background IRs of VTE reported from 11 data sources spanning six EU countries and derived from two observational studies, by focusing on the database as a source of heterogeneity. Through investigating data source characteristics potentially introducing differences in estimated IRs, our aim was to investigate the amount of unwanted (i.e., methodological) heterogeneity or uncertainty between data sources to provide more valid conclusions for safety surveillance activities. Data sources used in this study were mostly from primary care settings, partly with linkage to hospital data. The study used aggregated background IRs of VTE, considered a relevant AESI for a class of EU-approved COVID-19 vaccines.

Substantial heterogeneity in the background IRs was observed between all included data sources, in addition to observed within-data-source differences across age groups and genders. Age was the main contributor to the heterogeneity as shown in our study. Overall, it was observed that background rates increased with increasing age with no clear pattern in IR between males and females. The observation of increased IRs with increasing age is in line with another study on VTE,²⁹ with the same study also suggesting a difference in IR between genders: IRs increase markedly with age for men and women; the overall age-adjusted IR is higher for men (130 per 100 000) than women (110 per 100 000). The observed heterogeneity in the different age–gender strata is a source of information that leads to a better understanding of the burden of VTE in the general population. However, as demonstrated through the summary estimate and CIs, we still found substantial heterogeneity between data sources within each stratum, suggesting that there still might be unobserved patient-level heterogeneity and therefore a single estimate for each stratum might be inaccurate.

In an attempt to understand the contribution of database characteristics to the reported heterogeneity, we performed several exploratory analyses. Our databases included data derived from both hospital and primary care settings. In all data sources, when estimating background rates, it is important to consider how the population denominator was derived. When linking data between the two settings, depending on the mechanism of linkage, there is a risk of only capturing those subjects that had a hospital visit recorded, which could lead to biased estimates. Restricting the databases that included a link with hospital (PC-H linkage) resulted in a moderate decrease in the reported variability. Alongside, we did not see a decrease in heterogeneity using only databases that used the ICD-10 vocabulary demonstrating that the type of vocabulary used for clinical classification of VTE could not be identified as a major source of heterogeneity. Differences in background rates remained between the two studies even if the time at risk in which the rates were collected and age–gender subgroup definitions and analytical methods were similar. Comparing more closely the methodology applied in the two studies, differences in case definitions were noted, with some clinical codes only included in one of the studies (Table A1 in the appendix). The inclusion and exclusion criteria for individuals also differed, leading to non-identical study populations even when within the same data source. Since we did not find any systematic differences in estimates between the two consortia, it is unlikely that differences in case definition or inclusion criteria had a large influence on observed heterogeneity.

When quantifying heterogeneity using the statistical measure I², a considerable amount of heterogeneity (close to 100%) is reported. The large values of I² are not surprising, as the large sample sizes in every database imply small variance estimates. In particular the I² estimates in the 0–19 age group seem to be influenced by this fact: due to a larger sample size, the variance is lower than for the other age groups, leading to I² estimates that appear too high in comparison with the other age groups when looking at the forest plot.

In this study, we have calculated pooled estimates for the primary analyses. However, when large heterogeneity is present, focusing on a pooled estimate is not advisable, given that the pooled estimate will derive largely from the particular choice of databases and the relative weights associated with each database.³⁰ Following the classification in Deeks et al.⁸ it is not advised to combine the estimates if the value of I² was estimated to be larger than 90%. In addition, when reporting the results after combining estimates, attention needs to be given to uncertainty quantification. In addition to the common risk of misinterpretation for CIs,³¹ CIs for random-effects meta-analyses are easily misinterpreted to quantify dispersion of study effects. However, CIs only represent the uncertainty in estimating the mean effect size, not taking into account variability due to different database characteristics. This means, that CIs are always smaller than the range of observed estimates.

A major strength of this study is the use of data aggregated from a large number of data sources independently provided by two research consortia using the same calculation method to estimate the IRs. This enabled the exploration of database-specific aspects related to heterogeneity. From 11 databases, 54 257 284 subjects contributed to the main analysis; with the databases spanning a large part of Europe, this can be considered a representative sample of the total population.

The above exploratory analyses show that even when certain database characteristics are harmonized, significant heterogeneity is still present. A limitation of our study is that only aggregated data was available which prevented us from investigating potential sources of heterogeneity attributable to patient characteristics. For instance, comorbidities may have an influence on IRs.³² A final limitation is that clinical validation of the diagnosis codes was not performed in these studies which might have led to different frequencies of misclassification or underreporting of VTE cases, which may affect estimates of heterogeneity.³³

This study highlights the challenges regarding the varying levels of available information about database characteristics and the difficulty to identify sufficiently detailed information about the data sources. For example, some differences can only be explored through subject matter expertise about the corresponding health-care systems. Health-care systems might differ between regions, implying possible differences in the probability of recording certain events even in the same health-care setting. The process of clinical coding could also influence the quality of recording, with different levels of quality control or incentives for correct coding. With the large level of observed heterogeneity, an important recommendation is to use the same databases when comparing estimates at different time-points, for example, for pre- or post-exposure IRs of AESI.

The above considerations highlight the necessity for careful assessment of the suitability of databases to include in multi-database studies. In the two studies, a variety of databases was included because many different AESIs were considered simultaneously. For a study on a specific outcome, more specific restrictions on the databases should be placed a priori. In our study we have observed that the type of data source is one of the most important considerations. Based on subject matter knowledge or available validation studies, it should be evaluated in which type of setting the most accurate estimation is possible.

Our analysis also highlights the importance of unified case definitions and inclusion and exclusion cohort criteria to select adequate data sources in multi-database studies. This is evidenced in this study by the CPRD data source used by both study groups to address the same objective, but with a difference of 17% in the total number of individuals included in the study cohort, most likely related to differences in the operationalization of case definitions.

It is important to note that the methods used for detecting and addressing heterogeneity should be specified before starting any meta-analysis. When the forest plot show outliers among observed rates, it can be tempting to exclude the corresponding databases from the analysis without further investigating causes for outliers. This practice is, however, likely to introduce bias and should be avoided in most situations. Criteria for excluding certain databases should be specified prior to performing the analysis, but even then, it is advisable to also present results with the excluded databases, as a sensitivity analysis. In parallel, the choice of method for handling heterogeneity should also be prespecified, conditional of the outcome of the method for detecting heterogeneity. Also, it is preferable that the method for estimating the meta-analysis model and its statistical heterogeneity (e.g., REML), the methods for quantifying CIs (e.g., the Hartung-Knapp and Sidik-Jonkman modifications to the Wald method) and prediction intervals are prespecified.³⁴

More specifically, exploring the level of heterogeneity using multiple databases must be considered if these rates are intended to be used to support safety signal detection activities and to avoid misleading recommendations. One of the current initiatives is the DIVERSE project with the aim to develop guidelines for the identification, collection and reporting of heterogeneity in multi-database studies.³⁵ In addition, EMA's list of metadata for Real World Data catalogues,³⁶ which will be the basis of a catalogue of RWD sources, will provide researchers with standardized, relevant information about databases to use for RWE studies. Another approach would be to develop a set of metrics to measure database heterogeneity or to develop phenotype libraries to identify important variables in different databases. For instance, Ostropolets et al.²² quantify factors influencing IRs through a set of sensitivity analysis using patient level data. Finally, the use of SNOMED CT (systematized nomenclature of medicine – clinical terms),³⁷ a terminology that can cross-map to other classifications and code systems, may reduce the variability among estimates derived from different data sources. Nonetheless, the mapping of original coding systems to SNOMED may not reduce the heterogeneity as such, but may merely conceal possible heterogeneity introduced by different classification systems to operationalize the case definition of VTE. This is evidenced in this study by the ERASMUS data sources, where still large variability in the estimates is seen even when converted to SNOMED.

5 CONCLUSION

Our study revealed large variability in estimated age–gender-stratified background IRs of VTE between different databases, demonstrating the presence of one or several sources of heterogeneity. Restricting the databases to similar health-care settings contributed to less variability in the reported rates. Still, variability was present, triggered most likely by presence of analytical heterogeneity through differences in the case definitions and population cohorts as defined in the protocols used by the two study groups. The use of HARPER (harmonized protocol template to enhance reproducibility)³⁸ to operationalize code definitions will improve the creation of unambiguous clinical codes in studies integrating data from multiple data sources.

Our study can be utilized to better understand the complexity of RWE and to illustrate the importance of a cautious selection of databases, based on their characteristics, so that the observed heterogeneity represents true differences, to ultimately improve the reliability of RWE. Our findings should be considered in context of similar analyses with other databases and in other settings.

AUTHOR CONTRIBUTIONS

All authors meet ICMJE criteria, they have all contributed to the development of the study, as well as writing and approval of the manuscript.

DISCLAIMER

The views expressed in this article are the personal views of the authors and may not be understood or quoted as being made on behalf of or reflecting the position of the European Medicines Agency or one of its committees or working parties.