2.1. The Sample Formation

This retrospective study analyzed data of AIS patients, treated with intravenous alteplase, at the Neurology Clinic of the University Clinical Center of Vojvodina (Novi Sad, Serbia), during a 14-year period (2008–2022). The study was approved by the Local Ethics Committee (Approval Number 00-4, Date: 13 January 2023). The data were collected from the Clinical Information System, and the inclusion criteria were as follows: (I) A patient was older than 18 years; (II) the patient had no history of stroke; (III) intravenous alteplase was used as a treatment option; and (IV) a 90-day functional outcome, expressed as a modified Rankin scale (mRS) score, was known. On the contrary, ischemic stroke patients who were not treated with alteplase, or without known outcomes, were excluded from the study.

2.2. Statistical Analysis

The division into the outcome groups was made by the value of the 90-day mRS, and the groups were as follows: (I) patients with favorable outcomes (mRS ≤ 2) and (II) patients with unfavorable outcomes (mRS ≥ 3). The previously described division was adopted to maintain feasibility and align with the methodology of earlier research involving AIS patients (see Table 1).

Data processing was carried out in Python ver. 3.10.6 (Python Software Foundation, Wilmington, Delaware, United States) [35]. The first step was screening variables for missing values and excluding them from further analysis if this number exceeded 20%. Categorization of the variables followed, in which the division was made into categorical and continuous groups. During imputation, the most common value method was used for categorical variables, while the median value was used for continuous ones. Forty-nine variables were analyzed using adequate tests. Categorical variables were expressed as numbers (percentages), and the chi-squared test was used to determine differences between the two groups. Continuous variables were presented as medians (interquartile range) and analyzed by Student t-test or Mann–Whitney U test, based on sample normality. Two-tailed tests were used, and statistical significance was observed at level p < 0.05, for every variable. For continuous data normalization, Z-score was used to reduce numerical instabilities between the analyzed features [36].

2.3. ML Model Build-Up

The sample was described with 49 features. Based on the time point at which they were collected, the features defined four datasets: (I) baseline data, (II) 2-h data, (III) 24-h data, and (IV) discharge data (Figure 1). Based on this division, models encompassing different time frames were constructed: (I) The baseline model included baseline data; (II) the 2-h model included baseline and 2-h data, (III) the 24-h model included baseline, 2-h, and 24-h data; and (IV) the discharge model included all four datasets. The whole process and the research pipeline are summarized in Figures 1 and 2.

The dataset was split into a training set (80%) and a testing set (20%), by random splitting. The testing set was used to evaluate model performance. The training set was used for the feature selection process, which was carried out using the Least Absolute Selection and Shrinkage Operator (LASSO). This method is a regression analysis algorithm, which is often used to minimize potential collinearity, reduce a high-dimensional feature space by excluding noninterference variables, and reduce the overfitting of variables, which improves both prediction accuracy and interpretability [24, 37, 38]. The regularization parameter (alpha) was set at 0.2. Following the methodology of another study [13], 10 features with the highest importance were selected and used for the model training for every time point. The time point–based division of features along with their feature (patient data, clinical data, laboratory data, neuroradiological assessment, vascular risk factors, treatment data, stroke data, outcomes, discharge data, and hospitalization data) and variable (categorical or continuous) types is shown in Table 2. Features’ description is presented in Table S1.

2.4. Models’ Evaluation

The AUC measures the ability of a classifier to distinguish between classes. It is derived from the receiver operating characteristic (ROC) curve, which plots the true positive rate (TPR) against the false positive rate (FPR) at various thresholds. AUC values range from 0 to 1, where 1.0 indicates perfect classification, 0.5 indicates random guessing, and < 0.5 suggests poor performance (worse than random) [42]. Since it was proven that AUC is a better measure than accuracy in comparing learning algorithms [43], we used it to determine the best performing model.

Calibration is a degree of compliance between the probabilities predicted for each class and the accuracy of classifier on that prediction [44]. In our research, the Brier score was used as a measure of calibration. It should be noted that, unlike other metrics, a lower Brier score indicates better performance, that is, better model calibration [45].

2.5. Interpretative Framework

SHAPs represent one of the state-of-the-art ML interpretability models with a high extendibility. It is capable of calculating the contribution of each feature to the predicted outcome and visually represents it as the importance ranking [24, 46, 47]. SHAP can help us identify the output of the given classifier and understand the ML model decision-making process as valid and justified [20]. In this research, the models were represented using bar plots, which showed the degree of contribution, and violin plots, which visualized the overall correlation and directionality between features and the SHAP value [48].

LIMEs represent another interpretable model, which has shown itself useful in explaining individual samples or cases [24, 49]. This model is aimed at explaining the prediction-making process of any black-box model, by training a simpler interpretable model on a local subset of data around the example being explained [50]. The graph used in this analysis shows the overall predicted probability of a specific outcome on the left, decision-making process details in the middle, and the features’ values and categories on the right.

2.6. Dense Artery, mRS, Age, Glucose, Onset-to-Treatment Time, and National Institutes of Health Stroke Scale (DRAGON) Score

Built models were compared mutually and with a previously established prognostic score—the DRAGON score. The DRAGON score, a 10-point measure scale, has proven itself reliable in supporting clinical decisions. Encompassing six baseline variables (hyperdense cerebral artery CT sign, prestroke mRS, age, baseline glucose level, onset-to-treatment time, and baseline NIHSS), the score showed high predictive power for both good (mRS score 0–2) and miserable outcomes (mRS score 5–6) [51, 52].

Finally, the ROC curve was plotted, with TPR on the y-axis and FPR on the x-axis.

3.1. Sample Analysis

Our sample included 355 patients, with a median age of 67 years (IQR 60–74 years), 66% being male, and a majority (n = 196) having favorable 90-day outcomes. Patients with favorable outcomes were younger (65 years (IQR 57–70) vs. 71 years (IQR 63.5–77.0), p < 0.001), with lower values of baseline, 2-h, 24-h, and discharge NIHSS. They were hospitalized for a shorter time (12 days (IQR 7–15) vs. 15 days (IQR 10–22), p < 0.001), and their in-hospital stay was more commonly without complications (88% vs. 59%, p < 0.001). Described groups were compared, and the results of the statistical analysis are summarized in Table 3. The presented results are solely used to describe the sample and were not included in further analysis.

3.2. Classifiers

A complete table showing the best parameters for SVM, LR, and RF obtained from grid search is given in Table 4. In the case of SVM classifier, other than the linear SVM classifier, during the grid search process, the radial basis functions (rbfs) and sigmoid kernel were also considered. However, the results confirmed that the linear SVM was indeed the optimal choice for all the analyzed datasets.

The parameters [53] tuned via grid search for the SVM model included C, kernel, and gamma. The parameter C serves as a regularization parameter for controlling the penalty for misclassification. The kernel function transforms the input data into a higher-dimensional feature space, where the data becomes linearly separable. The gamma parameter, a coefficient of the kernel function, affects the decision boundary only if the chosen kernel function is “rbf,” “poly” (polynomial), or “sigmoid.” In the grid search, gamma was tuned alongside other parameters. However, since the optimal kernel identified for the model was “linear” in all cases, the gamma parameter had no impact on the model’s performance.

For the LR models, the tuned parameters [54] were C, solver, and class weight. Similar to the SVM, parameter C controls the regularization strength. The solver specifies the algorithm used to solve the optimization problem, aiming at finding the best possible fit for the training data. Class weight is used to deal with imbalanced class proportions by assigning different weights to different classes. When class weight is set to “none,” all classes are treated equally, and each class is given the same importance when the SVM model is trained.

For the third classifier, RF, the tuned parameters [55] were as follows: number of estimators, max depth, min sample split, min sample leaf, max features, and criterion. As its name suggests, the number of estimators refers to the number of decision trees (estimators) in the RF. The remaining parameters apply to each individual decision tree. The max depth parameter helps prevent overfitting by limiting the number of nodes in each decision tree [56]. Setting the max depth parameter to “none” means that there is no limit on the depth of the individual decision trees in the forest. Min sample split specifies the minimum number of samples required to split an internal node during the tree-building process. Min sample leaf defines the minimum number of samples required to be at the terminal node (leaf node). Max features determine the maximum number of features to consider when searching for the best split. If max feature parameter is set to “sqrt,” it means that square root of the total number of features will be used as maximum. Finally, the criterion parameter defines the function used to measure the quality of each split.

3.3. Model Evaluation

The evaluation metrics of models derived in 10-fold CV performed on a training set are given in Table 5, and metrics derived on a test set are given in Table 6.

On the testing set, several measures were used to give a comprehensive understanding of how well the classifiers were performing. For all classifiers, performance generally improved from the baseline to the discharge model, except for the relationship between the baseline and the 2-h models, where the values were slightly smaller in the second time point. SVM appeared to be the best overall classifier, with the highest accuracy, precision, sensitivity, F1 score, and AUC in most time points. LR demonstrated strong sensitivity (0.879 in the discharge model), but its performance in accuracy and precision was generally surpassed by SVMs and RFs. RF performed best in terms of sensitivity in the 24-h and discharge models and achieved the best Brier score in the discharged model. However, it had lower metrics in the baseline and the 2-h models, particularly in sensitivity.

Ultimately, AUC parameter was chosen for classifier comparison, and it was determined that the best classifier at each time point was as follows: SVM as the best baseline, 2-h, and 24-h model and RF as the best discharge model. The ROC curves calculated for the test phase are shown in Figure 3. The best models were further used for interpretable analysis. The models’ reliability diagrams are shown in Figure 4, and confusion matrices are presented in Figure S1.

3.4. Interpretation Analysis—SHAP and LIME

The decision-making process of the best-performing classifier, for each time point, was visualized using interpretable packages. Results of the SHAP analysis, as a feature-level interpretation method, are visualized in Figure 5, while the results of an individual-level interpretation model, or LIME, are shown in Figure 6.

3.5. Comparison of Models

For model comparison for different time points, AUC scores were used as an evaluation metric. The model based on the DRAGON score showed an AUC value of 0.760 (95% confidence interval (CI), 0.640–0.862). When compared, trained models outperformed the model based on the DRAGON score, and the difference became even more pronounced when compared to the 24-h and discharge models (Figure 7).

4.1. Downsides of Preconstructed Predictive Models and Scores

In the context of precision medicine in stroke care, a more individualized approach is needed to provide a personalized outcome prediction [57]. Using predominantly admission data, the medical community has created multiple scores that forecast a patient’s functional outcome. Some notable examples are Acute Stroke Registry and Analysis of Lausanne (ASTRAL), DRAGON, and Totaled Health Risks in Vascular Events (THRIVE). The resulting statistical models are considerably simplified to create integer-based scores since these scores are intended to be easily calculated by people using admission data. Consequently, the number of covariates is artificially decreased, and the weights of the models are discretized, which potentially worsens the models’ performance [19]. Besides these, some other multivariable models have been developed and validated. In most of the models, routinely collected features, such as age, sex, stroke severity, and comorbidities (e.g., atrial fibrillation and diabetes mellitus), consistently appeared as predictor variables of stroke patients’ functional outcomes. Even though methodological improvement and better model performances have been observed in previous years, if these models were externally evaluated, the discriminative power in terms of the 90-day favorable outcome (mRS ≤ 2) prediction was inconsistent (0.60 (95% CI, 0.57–0.64)–0.94 (95% CI 0.91–0.96)), raising concerns over reliability and usability of the models, as well as failure to assess clinical impact [10, 58]. It is also noteworthy that the majority of the models were developed in populations from developed countries [10], and that, due to possible racial and ethnic differences of the groups, various patients’ backgrounds, disparities in the healthcare system, hospital type, and/or available acute stroke treatment, although valuable, predefined prognostic models may not be the best option in all cohorts of AIS patients [2].

4.2. Adaptability of ML Models to the Dynamic Clinical Environment

After experiencing a stroke, the patient’s outcome depends on the various factors linked to different time frames of the recovery, and in a real clinical scenario, timing is another significant factor that influences treatment decisions [8, 59]. Therefore, prediction in only one time point may not be the most appropriate way to exploit the real power of the predictive models. In literature, the dynamic approach was more commonly used in patients treated with MT, as variables were divided into pre- and postinterventional datasets, observing higher predictive power when both types of features were included [14, 33]. In our study, we have focused on some of the most commonly used time points, as described in the previous study [19]. However, it is questionable whether these are the most appropriate time points since the real definition, timing, and duration of the critical period in stroke patients’ recovery are still unknown [60]. This represents a novel opportunity for the use of dynamic models, as, potentially, they could not only help us in the identification of this period but also guide us through it, resulting in a higher degree of patient functionality. The most important potential benefit of dynamic prediction is a timely adjustment of the treatment, possibly improving patients’ functional outcomes. One of ML’s main advantages is its ability to automatically learn from and adapt to changing settings. It can change without requiring the system designer to anticipate and address every possible scenario, which is particularly useful given the difficulty of a consistently changing and dynamic clinical environment [61–63]. Therefore, future studies could include more time points, resulting in a more precise prediction system.

4.3. ML-Based Model Interpretability as the Missing Link for Clinical Application

The usage and application of ML models were enhanced by the improvements in computational power, data availability, and dimensionality, which led to an increased interest in these systems. The models, although optimized for task performance and factors like safety, fairness, reducing technical debt, and providing explanations, when used, carry a risk of new legal and ethical issues [64–66]. Due to a sharp increase in ML model usage, a critical moment for ML in medicine has been reached, and a need for developing a methodological standard has emerged [65]. In 2024, the updated version of Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) was published, additionally including models based on ML (TRIPOD + AI). It offers a comprehensive item checklist stating reporting recommendations aiming at promoting complete, accurate, and transparent reporting of research based on the previously described models [67]. Although extremely valuable, this guideline has not included the interpretation of models, which is one of the main conditions for their clinical usage. Interpretability, on its own, is a poorly defined concept, and there is little consensus on its usage in ML [64, 68, 69]. It is described as the ability to trace back how ML models generate their results, as well as explain and present them in understandable terms to a human [64, 70, 71]. Interpretation is described as relevant if it sheds light on a selected domain problem for a specific audience [68]. The interpretable models showed potential in overcoming the incompleteness of the results in terms of limited understanding of the problem, safety, and ethics, leading to a better scientific understanding and certainty when making decisions [64, 72]. For patient consent and well-informed treatment decisions, it is essential to adequately understand the input–output relationship of a model, and in this scenario, the model interpretability is one of the primary obstacles identified to the widespread implementation of these methodologies [65, 72]. Since the reasoning behind the model’s behavior is crucial when making a decision, both for clinicians and patients, interpretability could ensure trust in the model, facilitating its clinical applicability [65, 66, 68, 69, 72, 73]. For this purpose, model-agnostic explanation methods such as SHAP and LIME have been developed, and they represent two commonly used techniques for model behavior analysis, as well as visualization of feature interactions and importance [65, 74–76]. Interpretable models demonstrate the surprisingly high utility of straightforward but precise models in practical applications, and they are essential for maintaining trust [73]. Based on everything said, we highly advocate for dynamic, interpretable ML model usage, as a multi–timepoint approach, followed by adequate explanations, could potentially increase the quality of stroke patients’ care.

4.4. Similar Studies

Most of the studies mentioned in Table 1 used a single time point approach, which predominantly relied on admission features, restricting their capacity to adapt to the dynamic recovery process in stroke patients. Although this approach produced good predictive results, studies that included additional time points into their models, such as those investigating patients treated with MT [13, 33], achieved better AUC values compared to the baseline-only models from the same studies. This suggests that a multi–time point approach could provide a more comprehensive understanding of patients’ recovery and could yield superior results. In addition, several studies done in both IVT- and MT-treated patients employed SHAP for interpretability [26, 31, 32]. While this method excels at feature-level interpretation and highlights overall importance in prediction, it does not provide patient-specific insights. For clinicians, understanding the reason why a specific patient is categorized into a particular outcome group still remains crucial for trust. As demonstrated in our study, LIME is capable of providing patient-specific interpretability, addressing the previously mentioned challenge. This capability highlights its potential and emphasizes the need for its broader adoption in predictive modeling.

4.5. Study Limitations

This study has several limitations. First, as the main goal of this study was to test a concept, the number of participants is relatively small. Therefore, a larger sample size is needed to validate these models for use in a real clinical environment. Second, models can predict the outcomes of only alteplase-treated patients, meaning that they cannot be used in patients treated with other modalities, such as tenecteplase or interventional approaches. Third, future models could implement more advanced techniques, such as DL, to exploit raw neuroradiological data and by that increase the predictive power of the models. While designing the study, we considered using DL models. However, DL is particularly advantageous for large, high-dimensional datasets, such as those involving images, text, or audio [77]. In contrast, for smaller, low-dimensional datasets, shallow ML models have been shown to outperform DL models and are generally easier to interpret [71, 78]. Given that our dataset is relatively small and does not include radiological images, ML models were deemed more suitable for this study. Fourth, it is worth mentioning that a central question during a supervised ML model build-up process concerns the accuracy of the resulting model and that a key problem is overfitting. Ideally, to prevent this problem, the model would be evaluated using new data originating from the same population. However, in practice, this is usually not feasible, and therefore, resampling methods, such as CV, play an important role in overfitting prevention [79]. Even though, in this study, the CV method was used in the analysis, more data should be collected, and the predictive model certainly needs to be evaluated in a real clinical environment.

4.6. Contribution and Future Directions

The main contribution of our research and proposed method lies in the comprehensive multi–time point utilization of routinely documented clinical data to gain valuable insights into the prognostics of AIS patients. The nature of the data used and the transparent understanding of the decision-making process make the models potentially implementable in everyday practice, provided the system is integrated with the hospital information system. However, we must acknowledge that this is a proof-of-concept study, and further investigations are needed to validate and build upon these findings. In the future, more complex models could be developed to directly incorporate CT/MRI images into the classifier, which, based on previous research, could further enhance outcome prediction.