Prediction of 90 day mortality in elderly patients with acute HF from e-health records using artificial intelligence

Study design

This study was conducted over the period from 1 January 2015 to 31 December 2021. The research was conducted in two hospitals on retrospective data collected during hospital stays: Paris Saint-Joseph Hospital and Lille University Hospital. The study followed the French National Commission for Information Technology and Civil Liberties (CNIL) reference methodology MR004 and was approved by the institutional ethics committee (institutional review board number IRB00012157). Patients were informed of the study by mail and given the opportunity to refuse to participate.

Study populations

The derivation cohort included all individuals hospitalized for HF at Paris Saint-Joseph Hospital between 1 January 2015 and 31 December 2020. The evaluation cohorts included all individuals hospitalized for HF at Paris Saint-Joseph Hospital (evaluation Cohort A) and Lille University Hospital (evaluation Cohort B) between 1 March 2021 and 31 December 2021. HF hospitalizations were identified by the International Classification of Diseases, 10th Revision (ICD-10) HF codes (I500–I501) and by HF keywords in the discharge summary. Keywords were searched with a natural language processing (NLP) tool considering misspellings and negations. The NLP tool was validated on 100 randomly selected discharge summaries of patients hospitalized at Paris Saint-Joseph with an ICD-10 code for HF and verified by two HF specialists. The NLP tool confirmed HF hospitalization with 97% specificity. Patients with at least one of the following criteria were excluded: aged over 95 years, receiving palliative care, with documented cardiac amyloidosis, who died during the hospitalization or who expressed their opposition to the study (Figure S1). Every admission meeting the inclusion criteria was included, meaning that a given patient could contribute more than one admission to the analysis.

Data extraction and management

Data available up to 48 h after admission in the EHRs of both hospitals were extracted. Two cardiologists selected an initial set of 151 clinical and biological variables considered relevant and obtainable routinely by the two hospitals for the management of HF. A least absolute shrinkage and selection operator²¹ (LASSO) regression was then used to select the final 42 variables in the model. For example, calcaemia, phosphoraemia and eosinophil count were not extracted. One hundred and fifty-one variables from 11 categories were extracted: demographics; hospital admission mode; month of hospitalization; medications (based on the anatomical therapeutic chemical classification); biology; comorbidities; electrocardiogram (EKG); left ventricular ejection fraction (LVEF) determined by echocardiography; vital signs; clinical signs of HF; and HF aetiology (Figure S2).

Structured data were preferred but, when necessary, NLP tools were used to search for keywords. For discrete variables, the first values were retained and for continuous variables, the mean values were used. A programme was developed to automate the extraction and cleaning of the data and to ensure the homogeneity of the data between the derivation and the evaluation cohorts, and between the two hospitals calculated (Supporting information Methods). Only variables with less than 25% missing data were retained. Missing values were imputed using the variable-specific statistics: for continuous variables, missing values were substituted with the mean value, and for categorical variables, missing values were replaced with the most frequent value.

The 90 day post-discharge vital status was obtained from the hospital information system, supplemented by the database of the French National Institute of Statistics and Economic Studies (INSEE). Matching between the study population and the INSEE database was performed using the INSEEHOP²² open-source tool.

Model development and testing

Logistic regression was used to predict the probability of 90 day post-discharge mortality by creating a model that correlates EHR variables and mortality with an output between 0 and 1. The model was developed in the derivation cohort and then independently tested on the two evaluation cohorts. Learning was performed using five-fold cross-evaluation.

Variables significantly associated with 90 day post-discharge mortality in a univariate model were included in a multivariate logistic regression. Given the large number of variables and the resulting risk of multicollinearity and overfitting, a LASSO²¹ regression was applied, with a threshold of 0.08, set to maximize cross-validation performance. LASSO is a regularization technique that promotes sparse patterns and limits overfitting, by automatically selecting variables on the basis of a penalty term derived from the absolute values of the coefficients. The variables selected by LASSO were then validated by clinical experts (M. K., P. d. G.). The final model was built with 42 variables (Table S3). This selection of variables not only improved the robustness of the model, but also facilitated its integration into the hospital information system.

The performance of the logistic regressions was evaluated with the area under the curve (AUC) of the receiver operating characteristics (ROC) curve for each of the three cohorts. AUC was also determined in subgroups: according to LVEF (<50% vs. ≥50%), and according to the number of medications recommended for HF (0 or 1 medication vs. ≥2 medications). Pharmacological therapies included at least diuretics, renin–angiotensin system (RAS) inhibitors, beta-blockers, mineralocorticoid receptor antagonists (MRAs) (Table 1).

The coherence of predictions obtained with logistic regression was verified with calibration and Kaplan–Meier curves. To plot the Kaplan–Meier curves, the evaluation population of both cohorts was pooled and then divided into three subgroups according to the terciles of 90 day mortality prediction [≤8.0%; (8.0%–12.5%); >12.5%]. Patients with a 90 day mortality probability of less than or equal to 8.0% were classified in the ‘low-risk’ subgroup, those with a probability between 8.0% and 12.5% were classified to the ‘intermediate-risk’ subgroup, and those with a probability greater than 12.5% were classified to the ‘high-risk’ subgroup. Kaplan–Meier curves were plotted up to 6 months of follow-up to assess the consistency of the risk groups over time.

Statistical analysis

After testing for normality of distribution, data were presented as mean ± standard deviation (normal distribution) or median [interquartile range] (non-normal distribution); categorical variables were presented as number (percentage). Characteristics of the derivation and of the two evaluation cohorts were compared using unpaired Student's t-test or Mann–Whitney test for continuous variables and the χ² test for categorical variables. Performance of the prediction model was determined by calculating the AUC; 95% confidence intervals (CIs) for AUC values were calculated by performing bootstrapping resampling. Differences in survival curves were tested using log-rank. A P value ≤0.05 was considered statistically significant. All experiments were performed using the Python package Scikit-learn version 1.3.0.

The AUC of our model was compared with that of the PROTECT score to predict the 90 day mortality post-discharge in the study population. The PROTECT score was calculated without the uric acid variable, as it was not available in the study population.

Characteristics of the derivation cohort

The derivation cohort included 1733 patients (2257 hospitalizations) (Table 1). Mean age was 81 years. Half of the cohort had an ischaemic HF (44.9%) and nearly half of the cohort had a preserved LVEF (48.8%). Almost half of the cohort (44.0%) had been admitted to intensive care. The most common HF medications were diuretics (79.6% of hospitalizations), followed by beta-blockers (56.1%) and RAS inhibitors (43.3%). MRAs were used in 15% of patients. Due to the recruitment period of this cohort, only one patient was treated with a Sodium Glucose Cotransporter 2 Inhibitor (SGLT2i). The rate of comorbidities was high, especially hypertension (76.3%), atrial fibrillation (57.6%) and diabetes mellitus (36.3%). Patients in the ‘0 or 1 medication recommended for HF’ subgroup were more likely to have preserved LVEF than those in the ‘≥2 medications recommended for HF’ subgroup (53% vs. 40% respectively, P < 0.05).

Two hundred and twenty-three patients (12.9%) died within 90 days of hospital discharge (248/2257 hospitalizations, 11.0%). They were characterized by older age, higher levels of B-type natriuretic peptide (BNP), serum creatinine and pacemaker implantation rate, as well as lower LVEF and prescription rate of RAS inhibitors. Furthermore, these patients were more likely to have 0 or 1 medication recommended for HF (Table 1).

Only three variables had more than 14% missing data. Of these, aspartate aminotransferase (ASAT) and BNP were retained in the final model, despite missing values rate of respectively 24.15% and 14.18%. The other variables in the dataset had missing values of less than 10% (Table S2).

Characteristics of the evaluation cohorts

Evaluation Cohort A included 312 patients (348 hospitalizations) and evaluation Cohort B included 353 patients (388 hospitalizations). There were several statistically significant differences between Cohort A and Cohort B. Patients of Cohort B were characterized by younger age, higher levels of BNP, higher prescription rate of diuretics and MRAs, as well as lower prescription rate of beta-blocker and anticoagulant, and pacemaker implantation rate. In addition, these patients were less likely to have cancer, whereas ischaemic HF and the presence of an implantable cardiac defibrillator (ICD) were more common in cohort B (Table 1).

Performance of the prediction model

The 90 day mortality rates were similar in the three cohorts (11.0% of the patients discharged in the derivation cohort, 10.9% in the evaluation Cohort A, and 10.3% in the evaluation Cohort B) (Table 2). Across all cohorts, the mortality rate tended to be higher in the ‘LVEF <50%’ subgroup and in the ‘0 or 1 medication recommended for HF’ subgroup.

The prediction model had an AUC: 0.817; 95% CI (0.789–0.845), in the derivation cohort. Performance in predicting 90 day post-discharge mortality was consistent in evaluation Cohort A [AUC: 0.750; 95% CI (0.672–0.829)] and evaluation Cohort B [AUC: 0.723; 95% CI (0.644–0.803)] (Table 2 and Figure 1). Across the derivation and the two evaluation cohorts, model performance was better in the ‘LVEF < 50%’ and in the ‘0 or 1 medication recommended for HF’ subgroups (Table 2).

The calibration curves showed that there was a good consistency between predicted probability and actual probability (Figure S3). Moreover, the three risk subgroups (‘low-risk’, ‘intermediate-risk’ and ‘high-risk’) had different survival curves and 90 day mortality rate (3.8%, 8.4% and 19.4% respectively) (Figure 2).

The PROTECT model had an AUC: 0.710; 95% CI (0.676–0.744), in the derivation cohort, and an AUC: 0.687; 95% CI (0.591–0.783), and an AUC: 0.655; 95% CI (0.571–0.739) in the evaluation Cohorts A and B, respectively.

Study strengths

This study has several strengths. The model was developed using data from all hospitalized HF patients, regardless of LVEF, and validated in two independent cohorts from different hospitals, indicating the model's robustness for different HF populations. As 90 day mortality was obtained by matching with the French national mortality database, very few patients were lost to follow-up. The calibration of the model was assessed with concordance between predicted and actual 90 day mortality to ensure accuracy. NLP was used to complement ICD-10 codes and improve the accuracy of diagnostic and comorbidity information, as previously demonstrated in HF.³⁴

Study limitations

Several limitations need to be acknowledged when interpreting our findings. The model was developed using data from a single centre in France, and although it was subsequently validated on two independent and clinically different populations, the possibility of location and selection bias cannot be completely excluded. Our model requires therefore further external validation and will be tested in multicentre studies to confirm its robustness and applicability across different healthcare facilities and hospitals.

Although the model gives promising results, the lower limit of the CI, falling below 0.7, could indicate a limitation in the model's discriminative power. A larger sample size of the evaluation cohorts could contribute to strengthening the model's robustness as well as including other populations from different cities and countries.

SGLT2 inhibitors have been shown to significantly improve mortality in HF in several large outcome clinical trials. The timeframe of development of our study did not allow to enrol many patients treated by SGLT2 inhibitors, which were not yet widely used. They were approved for the management of HF in our country in August 2021. The relevance of our model will therefore need to be reassessed in populations treated by this new class of HF. Nevertheless, practices for HF management in the current datasets are more contemporary than those used in older HF risk models.

Our population was made of elderly patients. However, subgroup analyses suggested that the algorithm performed well for the age group <80 years [Cohort A AUC 0.795 (0.689–0.902) and Cohort B AUC 0.697 (0.588–0.805)]. Whether our model is relevant for a younger HF population needs therefore further studies.

The current model is specific to the setting in which the data informing the model were obtained, that is, the prediction of 90 day mortality after hospitalization for decompensated HF, and it does not address prediction of rehospitalization or longer-term mortality. However, the 6 month survival curves show consistency in risk prediction.