Machine learning validation through decision tree analysis of the Epidemiology-Based Mortality Score in Status Epilepticus

1 INTRODUCTION

Status epilepticus (SE) is a frequent neurological emergency associated with high mortality, with an incidence that varies from 7.6% to 39% in population-based studies for convulsive SE.¹ Identifying predictors of mortality due to SE is crucial to guide clinical management through individual risk stratification. Ideally, prognostic scores should be able to correctly identify high-risk patients who will die (positive predictive value) and, at the same time, discriminate them from patients at low risk who have a high chance of survival (negative predictive value). A reliable predictive model would therefore guide the physician in selecting the treatment strategies most appropriate to different clinical scenarios and related prognosis. This would be important to improve management of SE, by rapidly referring more severe cases to the intensive care unit or conversely, to avoid overtreating patients with good outcomes, limiting unnecessary overdosing or adverse events associated with aggressive treatments.^{2, 3} A predictive score should be used to stratify patients according to the risk of a prespecified negative or positive outcome, and to apply differential treatment strategies to different prognostic groups, to individualize treatment.

The Epidemiology-Based Mortality Score in Status Epilepticus (EMSE) was introduced in 2015 to include epidemiological or real-world data for the prediction of in-hospital death among SE patients.⁴ The aim was to develop a score with higher diagnostic accuracy than the Status Epilepticus Severity Score (STESS), which considered only the level of consciousness at SE onset, age, SE semiology, and history of previous seizures.^{5, 6} Unlike STESS, which weights predictors based on prior assumptions, EMSE was based on mortality rates obtained by large epidemiological studies. It includes four parameters: etiology, age (stratified by decade), comorbidity (according to the Charlson Comorbidity Index),⁷ and the worst electroencephalographic (EEG) pattern (lateralized periodic discharges, after status ictal discharges, generalized sharply and/or triphasic potentials, and spontaneous burst suppression). Overall, validation studies showed that, at a cutoff point of 64 or 62, the positive predictive value for in-hospital death of EMSE was 30%–67%, and its negative predictive value for survival was 94%–100%.⁸

Decision tree analysis is a statistical machine learning technique for exploratory analysis and data interpretation that can be used to generate predictive models with higher accuracy than those developed using logistic regression, due to the higher detection of random relationships that may go unnoticed with other methods.⁹ They are employed to identify and visualize the relationships between predictors, making predictions on a selected outcome.¹⁰

This study aimed to identify which EMSE parameters prove most useful for prediction of 30-day mortality in SE patients using a machine learning approach.

2 MATERIALS AND METHODS

3 RESULTS

A total of 698 patients with SE were included, with a 30-day mortality of 28.9% (202/698). The demographic and clinical characteristics of the patients with SE are reported in Table 1, whereas EMSE parameters are reported in Table 2.

The mean EMSE value in the entire population was 57.1 (SD = 36.3); it was lower in surviving compared to deceased patients (47.1, SD = 31.7 vs. 81.9, SD = 34.8; p < .001).

Using the CHAID technique, four parameters included in the EMSE score, namely etiology, comorbidity, age, and EEG, were able to segment the data according to the risk of 30-day mortality after SE.

The decision tree is shown in Figure 1.

Etiology was the most relevant predictor, followed by age, EEG (poor EEG patterns: lateralized periodic discharges, after status ictal discharges, generalized sharply and/or triphasic potentials, and spontaneous burst suppression), and comorbidity group B (hemiplegia; moderate or severe renal disease; diabetes with end-organ damage; any tumor, including leukemia/lymphoma).

The decision tree identified 19 leaf nodes with a risk of 30-day mortality ranging from 0% (Nodes 1 and 5) to 67.8% (Node 11).

3.2 Nodes with a high risk of 30-day death

The following group of characteristics identified patients with a high risk of 30-day mortality.

1. Node 11. This node includes SE due to metabolic disorders or anoxia of any age and with an unfavorable EEG pattern (risk of mortality: 67.8%).
2. Node 17. This node includes SE due to metabolic disorders or anoxia in patients with age > 70 years and absence of poor EEG patterns (risk of mortality: 63.3%).
3. Node 9. This node includes SE due to head trauma or sodium imbalance or acute cerebrovascular disease and the presence of EMSE comorbidity group B (risk of mortality: 56.4%).
4. Node 15. This node includes drug reduction/withdrawal, poor compliance, remote cerebrovascular disease, brain injury, alcohol abuse, cryptogenic, brain tumor, or CNS infection and age > 70 years and an unfavorable EEG pattern (risk of mortality: 54.8%).

The Kaplan–Meier analysis confirmed a significant difference in survival and the cumulative risk of death (Supporting Information) between patients classified into different nodes.

The EMSE was accurate in predicting 30-day mortality, with an AUC ROC of .782 (95% confidence interval [CI] = .747–.816). The AUC ROC of EMSE in predicting in-hospital mortality was .800 (95% CI = .766–.834). The AUC ROC for specific subgroups was the following: first-ever SE episode, AUC ROC = .747 (95% CI = .702–.792); recurrent SE episode, AUC ROC = .827 (95% CI = .769–.885); nonhypoxic SE, AUC ROC = .764 (95% CI = .724–.803); and hypoxic SE, AUC ROC = .563 (95% CI = .429–.697).

The decision tree analysis using EMSE variables correctly predicted the risk of mortality in 77.9% of all cases; the prediction was accurate in 85.7% of surviving and in 58.9% of deceased patients within 30 days after the SE.

The decision tree in the subset of patients without hypoxic SE is shown in Figure 2, whereas Tables 3 reports demographic and clinical characteristics of these patients, and Tables 4 provides their EMSE parameters. The most relevant predictor was age, followed by EEG (poor EEG patterns: lateralized periodic discharges, after status ictal discharges, generalized sharply and/or triphasic potentials, and spontaneous burst suppression), and EMSE comorbidity group B (hemiplegia; moderate or severe renal disease; diabetes with end-organ damage; any tumor, including leukemia/lymphoma). The decision tree identified 13 leaf nodes with a risk of 30-day mortality ranging from 2.8% (Node 4; i.e., age < 60 years and absence of poor EEG patterns) to 62.1% (Node 11; i.e., age > 80 years and presence of EMSE comorbidity group B). In this subgroup of patients (SE other than hypoxic), the decision tree analysis using EMSE variables correctly predicted the risk of mortality in 78.9% of all cases; the prediction was accurate in 89.6% of surviving and in 46.1% of deceased patients within 30 days after the SE.

TABLE 3. Demographic and clinical characteristics of patients with nonhypoxic SE

Demographic and clinical characteristics	Survivors	Nonsurvivors	p
Patients, n (%)	469 (75.5)	152 (24.5)
Sex, n (%)			.390
Male	176 (37.5)	63 (41.4)
Female	293 (62.5)	89 (58.6)
Age, years, mean (range)	71 (59–79)	82 (72–87)	<.001
SE semiology, n (%)			<.118
GCSE	90 (19.2)	20 (13.2)
FCSE	143 (30.5)	42 (27.6)
NCSE	228 (48.6)	89 (58.6)
MSE	8 (1.7)	1 (.7)
NCSE in coma			<.001
Yes	47 (20.6)	39 (43.3)
No	181 (79.4)	50 (46.7)
Etiological classification according to ILAE, n (%)			.001
Acute symptomatic	255 (54.4)	110 (72.4)
Remote symptomatic	101 (21.5)	16 (10.5)
Progressive symptomatic	92 (19.6)	20 (13.2)
Cryptogenic	12 (2.6)	6 (3.9)
SE in defined electroclinical syndromes	9 (1.9)	0 (.0)
Refractory SE, n (%)	47 (10.0)	56 (36.8)	<.001
Superrefractory SE, n (%)	39 (8.3)	18 (11.8)	.198
Prior history of epilepsy, n (%)			.003
Yes	197 (42)	43 (28.3)
No	272 (58)	109 (71.7)

• Abbreviations: FCSE, focal convulsive SE; GCSE, generalized convulsive SE; ILAE, International League Against Epilepsy; MSE, myoclonic SE; NCSE, nonconvulsive SE; SE, status epilepticus.

TABLE 4. EMSE parameters in patients with nonhypoxic SE

EMSE parameter	Survivors	Nonsurvivors	p
Patients, n (%)	469 (75.5)	152 (24.5)
Age, n (%)			<.001
21–30 years [EMSE points: 1]	16 (3.4)	0 (.0)
31–40 years [EMSE points: 2]	23 (4.9)	2 (1.3)
41–50 years [EMSE points: 3]	30 (6.4)	4 (2.0)
51–60 years [EMSE points: 5]	64 (13.6)	6 (3.9)
61–70 years [EMSE points: 7]	98 (20.9)	17 (11.2)
71–80 years [EMSE points: 8]	133 (28.4)	40 (26.3)
>80 years [EMSE points: 10]	105 (22.4)	86 (56.6)
Etiology, n (%)			<.001
CNS anomalies [EMSE points: 2]	12 (2.6)	0 (.0)
Drug reduction/withdrawal, poor compliance [EMSE points: 2]	30 (6.4)	4 (2.6)
Multiple sclerosis [EMSE points: 5]	7 (1.5)	0 (.0)
Remote cerebrovascular disease or brain injury [EMSE points: 7]	155 (33)	27 (17.8)
Hydrocephalus [EMSE points: 8]	4 (.9)	0 (.0)
Alcohol abuse [EMSE points: 10]	4 (.9)	1 (.7)
Drug overdose [EMSE points: 11]	12 (2.6)	0 (.0)
Head trauma [EMSE points: 12]	16 (3.4)	7 (4.6)
Cryptogenic [EMSE points: 12]	26 (5.5)	7 (4.6)
Brain tumor [EMSE points: 16]	69 (14.7)	17 (11.2)
Metabolic: sodium imbalance [EMSE points: 17]	13 (2.8)	6 (3.9)
Metabolic disorders [EMSE points: 22]	41 (8.7)	42 (27.6)
Acute cerebrovascular disease [EMSE points: 26]	57 (12.2)	38 (25)
CNS infection: acute [EMSE points: 33]	23 (4.9)	3 (2.0)
EMSE comorbidity group, n (%)
Group A: myocardial infarction, congestive heart failure, peripheral vascular disease, cerebrovascular disease, dementia, chronic pulmonary disease, connective tissue disease, ulcer disease, mild liver disease, diabetes [EMSE points: 10]	215 (45.8)	117 (77)	<.001
Group B: hemiplegia; moderate or severe renal disease; diabetes with end-organ damage; any tumor, including leukemia/lymphoma [EMSE points: 20]	90 (19.2)	68 (44.7)	<.001
Group C: moderate or severe liver disease [EMSE points: 30]	4 (.9)	5 (3.3)	.044
Group D: metastatic solid tumor, AIDS [EMSE points: 60]	19 (4.1)	10 (6.6)	.192
EEG, n [%]			<.001
No LPDs, GPDs, or ASIDs [EMSE points: 0]	337 (71.9)	74 (48.1)
ASIDs, LPDs, GPDs [EMSE points: 40]	129 (27.5)	75 (49.3)
Spontaneous burst suppression [EMSE points: 60]	3 (.3)	3 (2.0)

• Note: The EMSE points were reported according to Leitinger et al. 2015.⁴
• Abbreviations: AIDS, acquired immune deficiency syndrome; ASID, after status ictal discharge; CNS, central nervous system; EEG, electroencephalogram; EMSE, Epidemiology-Based Mortality Score in Status Epilepticus; GPD, generalized periodic discharge; LPD, lateralized periodic discharges; SE, status epilepticus.

4 DISCUSSION

In this study, we have used the innovative decision tree analysis based on machine learning technique to evaluate the role of the EMSE in predicting 30-day mortality in a large cohort of patients. Decision tree analysis showed that EMSE parameters can be used to predict the risk of short-term mortality. It found that certain etiologies, comorbidities, and age groups predict survival, whereas others are associated with an increased risk of 30-day mortality. Our analysis, therefore, identified the most important discriminators to separate between survivors and nonsurvivors.

Our analysis confirms that older age predicts short-term mortality after SE, a finding that has been consistently reported in the literature.^{5, 19-22} Age was included also in the STESS, which dichotomized age as younger than 65 years or 65 and older.^{5, 6} In this regard, the EMSE scoring system provides different points per each decade, enabling a more individualized risk prediction. Actually, our analysis showed that age > 70 years predicts 30-day mortality.

An interesting aspect of this method is that it can predict the risk of death and survival by “weighing” a finite number of clinical variables that are easily applicable to the individual case. By doing this, it identifies some “clinical scenarios” that could be considered for individualized prognosis and counseling. For example (see Figure 1), a 75-year-old patient with an SE associated with remote brain traumatic injury will have a 30-day probability of death of approximately 30% (Node 7). However, if EEG monitoring documents an unfavorable pattern, the 30-day probability of death will increase to >50% (Node 15). The proposed decision tree, therefore, allows the clinician to simply and immediately determine which “box of risk” the patient is in at that moment. The first advantage is to communicate with patient's relatives and with other health care personnel by providing more precise information on the individual risk of death in this specific clinical scenario. Furthermore, an accurate prognostication could have an impact on management strategies, particularly when deciding about the aggressiveness of treatment or the need of referral to the intensive care unit with intubation/ventilation and administration of anesthetics, which might appear justified in cases with a more severe prognosis, but not in milder cases. Obviously, whether a predictive score may guide treatment remains an open issue, as treatment is not part of the score calculation. However, having accurate information on the overall prognosis appears relevant for clinical decisions related to the treatment of SE. According to the evidence-based practice framework, any clinical decision should rely upon the best available evidence, and be integrated with the clinical expertise of the treating physician and with the patient's and his relatives' perspective.

The groups of characteristics (nodes) associated with a good prognosis indicate scenarios in which a prompt and aggressive treatment of the underlying etiologies and the SE is required to maximize the chances of recovery. This does not mean, however, that cases associated with poor prognosis should not be adequately treated, because as confirmed in this study, the EMSE score is less accurate in identifying patients who die within 30 days from the SE onset than survivors.

Notably, our model takes into consideration only the EMSE parameters; therefore, it does not include in the machine learning analysis at least two other parameters that could be relevant to increase accuracy: the response to treatment with first/second-line drugs and the semiology of the SE. Both EMSE and STESS scores were evaluated for the prediction of refractoriness to first- and second-line drugs, showing that accuracy was poor for both scales.² Therefore, the response to treatment with first/second-line ASMs and short-term mortality should be specifically evaluated in future studies, and eventually incorporated in machine learning models. Regarding SE semiology, this has been recently evaluated by our group to explore how it clusters with other different SE features.^{23, 24} The vigilance level (assessed by the Glasgow Coma Scale) appeared to be a useful scale to stratify the alteration of consciousness during SE and was associated with different unfavorable predictors outcomes, as confirmed by a recent epidemiological study by Leitinger et al., who clearly showed how impairment of consciousness in the semiology evolution of an SE episode is associated to increased mortality.²⁵

Another relevant consideration concerns the EEG. EMSE, and therefore our model, clearly show the importance of identifying the “worst” EEG pattern. If it consists of certain patterns that are well-known predictors of poor prognosis, the EEG can modify the risk of death in the decision tree. This means that our model (like the EMSE) requires a careful EEG evaluation (and monitoring) of the patient. This poses some limitations in applicability, especially in an emergency setting when EEG is not available, or in situations with limited resources. Furthermore, because EEG is often performed after treatment start, at least in the case of generalized convulsive SE, some of its pattern may reflect and be influenced by treatment choice. In this regard, the development of similar decision tree models using the STESS score (which does not include EEG) could be a valuable option in the future to provide individual case stratification in the absence of EEG information or if it is suspected that EEG patterns reflect the treatment choice rather than SE intrinsic severity with different prognosis.

Finally, this study has a few other limitations. First, it was conducted at a single tertiary care center, possibly limiting the generalizability of its findings. However, the large number of patients included and the robustness of the statistical analyses strengthen the validity of the results. Rather than in-hospital mortality (the outcome that was adopted in the EMSE), we used mortality at 30 days after the SE.

The aim of the present study was to validate the EMSE, evaluating whether it was able to accurately predict short-term mortality following SE using a machine learning method. For this reason, we did not change the EMSE parameters, but assessed their prognostic accuracy in a validation study. For the same reason, we did not compare EMSE to STESS (with cutoff values of 3 or 4 for poor outcome) for the accuracy in predicting mortality. Few studies have compared the prognostic accuracy of these two scores,^{2, 26} and their findings should be replicated. Furthermore, future studies are required to develop prognostic models that consider other parameters not included in the EMSE (e.g., SE semiology or changes in SE semiology over time).

We used the decision tree analysis to validate the EMSE as a prognostic score for the entire population of SE patients, as it was originally conceived and developed. The predictive performance of the EMSE evaluated using AUC ROC was good, both for 30-day mortality and for in-hospital mortality (the outcome used for the development of EMSE); the latter was .800, a value consistent with other studies evaluating the predictive value of EMSE. Similarly, EMSE performance for mortality prediction in the subset of patients with nonhypoxic SE was good; the AUC ROC for this subgroup was high (.764, 95% CI = .724–.803), and the decision tree analysis using EMSE variables correctly predicted the risk of mortality in 77.9% of cases (in 85.7% of survivors and in 58.9% of nonsurvivors). However, the predictive performance of the EMSE could be suboptimal for specific patient subgroups, who may require more focused predictive models. As an example, the AUC ROC of the EMSE in predicting 30-day mortality for the subgroup of hypoxic SE was only .563 (compared to .782 in the entire SE population). This may be reasonably due to the EMSE including several parameters reflecting the intrinsic high heterogeneity of SE, and it therefore proved less effective in accurately predicting mortality in a highly homogeneous condition such as hypoxic SE. Furthermore, hypoxic SE is associated with a high risk of mortality, whereas the EMSE appears particularly accurate and effective in identifying survivors. The EMSE was developed to predict the prognosis of SE considered as a whole, and it should be used with this aim and in this population. Constructing models for various SE subgroups (e.g., according to clinical semiology or etiology, such as hypoxic SE) and evaluating their predictive performance, although outside the scope of the present study, could represent an interesting and promising approach toward a more individualized risk prediction (precision medicine).

5 CONCLUSIONS

Using the innovative machine learning technique of decision tree analysis in a large cohort of patients, our study has shown that EMSE is a valuable prognostic tool, and it appears particularly accurate and effective in identifying patients with 30-day survival. Its value in predicting 30-day mortality is, however, low and needs to be further improved.

AUTHOR CONTRIBUTIONS

Conceptualization: Francesco Brigo, Gianni Turcato. Methodology: Francesco Brigo, Gianni Turcato, Simona Lattanzi, Arian Zaboli. Writing–original draft preparation: Francesco Brigo, Gianni Turcato. Formal analysis: Gianni Turcato. Writing–review & editing: Simona Lattanzi, Niccolò Orlandi, Giulia Turchi, Arian Zaboli, Giada Giovannini, Stefano Meletti. Data curation: Niccolò Orlandi, Giulia Turchi, Giada Giovannini. Investigation: Niccolò Orlandi, Giulia Turchi, Giada Giovannini. Visualization: Arian Zaboli. Supervision: Stefano Meletti. Validation: Stefano Meletti.

CONFLICT OF INTEREST

None of the authors has any conflict of interest to disclose.