2.1 Developing the review questions

2.2 Conducting the search strategy

Initial search was conducted using the PubMed and Cochrane Library databases. The Medical Subject Headings (MeSH) used were: “cerebrovascular disease,” “magnetic resonance imaging,” “artificial intelligence,” and their associated terminologies. The search covered the period from January 1, 2018 to December 31, 2023.

2.3 Study selection

2.4 Data extraction

3.1 Input data

The input data used for the basis of the AI learning vary according to the clinical task, with models using the input data to learn associations between specific features and outcomes of interest. These data may include, but are not limited to, demographics, physical examination notes, and clinical images.

The distribution of sample sizes varied widely between the studies. Of the included studies, 50/67 used more than 100 dataset entries, 6/67 used more than 1000, and 6/67 assessed 50 or fewer participants. Among the eligible studies, 31/67 described a method validated on images obtained from multicenter datasets. Moreover, the majority of image data were collected from nonpublic sources (51/67). Among the studies using data collected from public sources, The Ischemic Stroke Lesion Segmentation Challenge was the most commonly used database (n = 5).

MRI has been widely used for brain analysis and nearly half of the proposed methods used multisequence scans as input (29/67). Various MRI modalities have previously been used for AI to analyze CVDs and extract phenotypic features, and the MR sequences used as input data in the reviewed studies are shown in Figure 3e.

3.2 Preprocessing

Intensity inhomogeneity, various artifacts, and noise pose challenges when automated analysis is applied to MRI and appropriate preprocessing methods are required to remove noise, eliminate irrelevant information, recover useful information, and enhance the detectability of relevant information, thereby reducing the complexity and computation time of image analysis and modeling. The preprocessing procedures are including registration (n = 25), N3/N4 bias field correction (n = 9), skull stripping (n = 24), spatial smoothing (n = 5), intensity standardization (n = 38), and filtering (n = 7) (for explanation of the preprocessing methods see Table S2). To solve these problems, various image processing programs have been applied to MRI images including FSL, BET, Robex, FreeSurfer, and SPM.

3.3 Modeling

Modeling, being the core of AI methods, involves extracting biomarkers for disease identification from the original source data and performing image processing (image segmentation and image generation). Computer-aided diagnosis (lesion detection, disease classification, and physiological index estimation) is then performed based on the extracted features (Figure 3f). Details of particular algorithm tasks are explained in Table S3.

The classic ML method requires the selection of appropriate features using feature extraction and selection methods, which rely on expert experience, whereas useful representations and features are learned automatically in DL. Various classic ML models (n = 14) were used for analyzing CVD including support vector machines (SVM, 50.0%), random forests (RF, 50.0%), logistic regression (LR, 42.9%), artificial neural networks (ANN, 14.3%), decision trees (DT, 7.1%), and k-nearest neighbor (KNN, 7.1%). The deep neural network architecture used in DL (n = 61) varied with the clinical tasks, with most studies using a CNN (93.4%) composed of convolutional layers, pooling layers, and activation layers such as ResNet, VGG, U-Net, and DeepMedic. Other DL methods including autoencoding (n = 1), natural language processing (NLP, n = 2), and recurrent neural networks (RNN, n = 1) were also used. Importantly, 7/67 studies combined ML and DL algorithms. The distribution of the AI methods used for diagnosing CVD is shown in Figure 4.

3.4 Performance evaluation

Performance measurement is an essential task when analyzing the feasibility of methods for clinical application. The benchmark measures used to evaluate the reviewed studies are shown in Figure 5.

To validate the generalization ability of the model, cross-validation methods were used to reduce variations and avoid bias within a single dataset. These included k-fold cross-validation (n = 34), nested k-fold cross-validation (n = 1), and leave-one-out validation (n = 1), methods that are suitable for small sample datasets, particularly for medial image analysis where the number of positive samples may be relatively low. External validation defines the extent to which the findings of a study can be generalized to other situations, people, settings, and measures. However, only 24/67 studies used external validation to evaluate their models.

3.5 Key research questions and clinical utility grading

We conducted an analysis of the literature regarding the application of AI to MRI of CVDs and identified and categorized the most critical research questions on the basis of the likelihood of their prompt integration into clinical practice. These questions were classified into the following three categories: A, imminent clinical applicability; B, promising potential for clinical translation; and C, unresolved challenges that impede practical implementation in clinical settings. The results are listed in Table 1.

4.1 Question 1. Which MR sequences are most commonly used as AI input in studies on the different types of CVD?

In recent decades, the application of AI to MR images of CVDs has increased substantially. Among the 67 articles reviewed herein, more than half discussed applications in stroke, while the others described an extremely extensive range of CVDs including ICA, CMB, and MMD. The MR sequences used in the applications also varied widely. Among the articles, the application rates for particular sequences were 49.3% for diffusion-weighted imaging/apparent diffusion coefficient (DWI/ADC), 26.9% for Fluid Attenuated Inversion Recovery (FLAIR), and 17.9% for T1-weighted imaging, rates that may reflect the more easily accessible clinical imaging data sources. These MR sequences are commonly used in clinical practice and are convenient for retrospective analysis. Notably, in some particular types of CVDs, the most advantageous MR sequences for conventional diagnosis are still the most commonly used such as DWI/ADC for stroke (33/40), magnetic resonance angiography for ICA (12/13), and susceptibility-weighted imaging for CMB (10/10). In recent years, functional MR techniques such as blood-oxygen level-dependent-functional MRI, diffusion tensor imaging, and arterial spin labeling have expanded the clinical potential of AI for CVD.

Of the 67 articles analyzed in this study, 51 (76.1%) were collected from nonpublic sources and 17 (25.3%) used fewer than 100 dataset entries. These findings may indicate the limited generalizability of the AI models. In data-driven analyses, image quantity and quality are crucial and decisive. Such small sample sizes are insufficient for clinical validation even if the images have originated from multiple scanners. The quantity and quality of datasets greatly impacts model performance, which will further affect the promotion of research results in clinical practice.

4.2 Question 2. What are the specific theories and tasks of existing algorithms?

The algorithms or models most commonly used in AI methods are the classic ML and DL algorithms. In 7/67 articles, both ML and DL were used in conjunction. There is currently no standardized reporting scheme for AI methods, especially DL models. AI researchers have always adjusted parameters to optimize the model. Although the performance advantage of the optimized model was demonstrated by comparing it with the conventional results in most studies, the clinical value of the model was still limited. From a clinician's perspective, the ultimate purpose of AI is to assist in clinical diagnosis, rather than developing increasingly complex models to confuse doctors [75]. Conversely, the goal of AI scientists is to maximize model training and perform continuous optimization. Therefore, increased collaboration between clinicians and AI scientists will promote the effective integration of AI into clinical practice.

In the included studies, AI algorithms were expanded to perform more complex tasks including image classification, detection, segmentation, generation, and estimation. The differences in algorithm tasks are mainly related to the study purpose and type of CVD. The overall rate of image classification and segmentation was nearly 76.1%, with stroke accounting for the highest proportion (40/51). Meanwhile, the overall rate of image detection reached 28.3%, and ICA and CMB accounted for the entire proportion (19/19). These results are associated with clinicians' requirements for AI-assisted diagnoses of different diseases. For stroke, clinicians are generally more concerned about AI-assisted extraction of key features to facilitate lesion classification and prognosis assessment. However, for ICA and CMB, the greatest application value of AI in clinical practice is to help identify lesions and avoid missing a lesion diagnosis. Moreover, the prediction of ICA rupture and differentiation of CMB etiology are important research directions for AI in the future.

4.3 Question 3. How many commonly used model validation methods were applied?

Considering the limitations posed by low numbers of positive medical image samples and the differences between datasets generated by different centers and scanners, external validation of models is recommended. While some external validations have shown models to have good generalization capabilities, studies have also identified a difference in performance between internal and external validations.

Many studies did not perform external validation of the model because of its time-consuming and labor-intensive nature, instead relying on the results from the internal validation data, which may lead to overestimation of the model performance. This overfitting problem effecting ML methods has been well-described. Many studies have used cross-validation to resolve the overfitting problem presented by internal validation. Among them, k-fold cross-validation has been widely used, with the choice of k being mostly between 5 and 10.

In summary, we conclude that external validation beyond a single dataset should be performed to evaluate model generalizability, and efforts should be made to reach statistical consensus across multiple datasets. Moreover, a cross-validation method should always be attempted as an internal validation for the proposed model, particularly in the absence of external validation. However, it should be noted that the proposed model may not be applicable to other unseen datasets even if external validation is performed.

4.4 Question 4. Do the most commonly used evaluation metrics vary according to different algorithm tasks?

Common model evaluation metrics include True Positive (TP), FP, false negative, true negative, accuracy, sensitivity, specificity, and receiver operating characteristic curve/ area under curve (ROC/AUC). The evaluation metrics selected may differ according to the different tasks. In the studies reviewed here, if the task was classification, ROC/AUC, sensitivity/True Positive Rate (TPR), specificity, accuracy, and TP were commonly used; if the task was detection, TP, FP, and sensitivity/TPR were commonly used; if the task was segmentation, the Dice similarity coefficient and Hausdorff distance were commonly used; if the task was generation or estimation, mean square error, root mean square error, and statistical analysis were commonly used. Our results indicate that a complex array of model evaluation metrics have been used, which might have confused clinicians and led to misinterpretation of the conclusions. Therefore, in future AI research, the evaluation metrics should be standardized and easy to understand to facilitate comparisons between different studies.

4.5 Question 5. What are the key questions of the research and how far are the proposed AI models from clinical practice?