2.1 Participants

This retrospective study was approved by The Ethics Committee of Huashan Hospital (KY2021-452), and the requirement to obtain informed consent was waived. Patients were recruited from Huashan Hospital, Fudan University, between August 2020 and September 2022. The inclusion criteria were as follows: (1) new onset or recurrent supratentorial glioma, (2) surgical treatment was performed (removal or biopsy), and (3) pathologic diagnoses were determined according to the 2021 World Health Organization (WHO) classification of CNS tumors. The exclusion criteria were as follows: (1) presence of brain metastasis, (2) extra-axial tumors, and (3) glioma involving the brainstem or the spinal cord. The study was registered in WHO International Clinical Trials Registry Platform (registration No. ChiCTR2000036816) and 15 glioma cases were randomly selected.

2.2 MRI and positron emission tomography imaging acquisition

MRI was performed within 14 days prior to surgery. Routine clinical sequences, including contrast-enhanced T1-weighted imaging (T1WI, repetition time [TR] = 6.49 ms, echo time [TE] = 2.9 ms, flip angle [FA] = 8°, spatial resolution = 0.833 mm × 0.833 mm × 1 mm), were acquired at 3T on an Ingenia MRI scanner (Koninklijke Philips N.V., the Netherlands).

Chemical exchange saturation transfer (CEST) acquisitions were performed on a 7T MRI scanner (MAGNETOM Terra, Siemens Healthineers, Erlangen, Germany). A prototype snapshot-CEST sequence (optimized single-shot gradient-echo [GRE] sequence with rectangular spiral reordering) was applied (TR = 3.4 ms, TE = 1.59 ms, FA = 6°, bandwidth = 660 Hz/pixel, grappa = 3, resolution = 1.6 mm × 1.6 mm × 5 mm). Z-spectrum readings were sampled between −300 and 300 ppm, with denser sampling between −6 and 6 ppm. A total of 56 frequency offsets were obtained after saturation pulses, using three different radiofrequency (RF) power levels of B1 = 0.60, 0.75, and 0.90 μT. Whole-brain high-resolution T1WI (MP2RAGE; TR = 3800 ms, TI1 = 800 ms, TI2 = 2700 ms, TE = 2.29 ms, FA = 7°, spatial resolution = 0.7 mm³ isotropic) and T2-weighted imaging (T2WI, SPACE; TR = 4000 ms, TE = 118 ms, spatial resolution = 0.67 mm³ isotropic) were acquired for registration [9].

Pulsed-gradient spin-echo diffusion-weighted imaging (DWI) data were obtained with TR = 4500 ms, TE = 56.8 ms, 1.5 mm isotropic resolution, and two shells of b-values (b = 1000 and 2000 s/mm²) with 64 directions for each b-value. All DWI data were corrected using denoising, top up, and eddy correction (MrTrix3, https://www.mrtrix.org/). Diffusion tensor imaging (DTI) parameters were estimated, including mean diffusivity, apparent diffusion coefficient (ADC), and FA.

Amino acid positron emission tomography (PET) scans were performed within 14 days prior to surgery using a Biograph 128 PET/CT system (Siemens, Erlangen, Germany). Patients fasted for at least 4 h, and 370 ± 20 mBq ¹⁸F-FET was intravenously injected before scans. The uncorrected radiochemical yield was about 35%, and the radiochemical purity was above 98%. The tracer was administered as an isotonic neutral solution. A static scan was performed 20 min after tracer injection and lasted for 20 min. Attenuation correction was performed using low-dose CT (120 kV, 150 mA, Acq. 64 mm × 0.6 mm, 3 mm slice thickness, 0.55 pitch) acquired before the emission scan. PET images were reconstructed using the iterative 3D method with a Gaussian filter (6 iterations, 14 subsets, full width at half maximum = 2 mm, zoom = 2) [10].

2.3 Pathology and isocitrate dehydrogenase mutation assessment

All patients received a pathologic diagnosis from neuropathologists made according to the 2021 WHO classification of CNS tumors. Isocitrate dehydrogenase (IDH) mutation status was tested using immunohistochemistry or sequencing.

2.4 Experimental design

We adopted ChatGPT-4V (https://chat.openai.com/) as the representative multimodal AI model to test with 7T imaging data. We manually selected the most typical slices containing the tumor's observable feature and converted them to 2D Portable Network Graphics images, which were then combined with different textual task-related prompts and input into ChatGPT-4V. For multimodal tasks or other cases, we combined all subfigures and input one integrated Portable Network Graphics image into ChatGPT-4V. Specifically, we designed four tasks to progressively test the different levels of ability that potentially underpin the clinical applications using multimodal AI and 7T imaging data (Figure 1). The first two tasks were to test ChatGPT's conceptual preparations, while the latter two directly tested ChatGPT in diagnostic applications. Data from 15 subjects were included in this research (the data sizes used varied in accord with the purpose of the different tasks), and the typical tumor slices were selected on the basis of the largest tumor diameter on T2WI. Before the start of all tasks, we initialized ChatGPT using a prompt such as “Assuming you are a medical school student currently taking an exam, here are your questions”.

In Task 1, we tested whether ChatGPT-4V has enough conceptual preparation for ideal usage of 7T data. We asked ChatGPT-4V for knowledge about the difference between the 3T and 7T images, using prompts such as “Do you have any knowledge about 7T MRI?” and “What is the major difference between 3T and 7T MRI?”. Furthermore, we required ChatGPT-4V to differentiate real 3T and 7T data based on its encoded knowledge, using pre-prepared 3T and 7T imaging data and a prompt such as “On the basis of your knowledge, can you recognize which subfigure is from 3T MRI and which one is from 7T?”. This test involved 16 3T and 16 7T images, which were input as two pairs in one round of question answering (Supplementary File 1). The spatial organization of the input images was shuffled to ensure that ChatGPT-4V was identifying the images from knowledge rather than from random guessing.

In Task 2, we provided 7T imaging data from a wider spectrum of imaging modalities, and investigated whether ChatGPT-4V could handle all of the modalities with its corresponding knowledge. We therefore tested whether ChatGPT-4V can recognize 7T imaging data from different modalities. In this study, we included T1WI, T2WI, CEST, quantitative DTI maps, ADC, FA, MD, and color-coded FA [colFA). Considering that CEST is not a common modality and other potential sources of confusion, we designed the prompt as “Above I provide you a series of 7T images, including MRI (including one CEST imaging data), ADC, and quantitative DTI map (such as FA, MD). Could you recognize the corresponding modalities of each given image?”. In Supplementary File 2, we provide the response to a version of the question with less hints.

If ChatGPT-4V passes the above two tests, it would show promise for the ability to assist in clinical tasks with 7T images. In Task 3, we simulated an image reading and diagnosis process using single-modality 7T imaging data (T1WI and T2WI data). ChatGPT-4V was instructed to identify the abnormality in the given 7T image and annotate the identified lesion in the input image. The prompt was written as “Can you tell what is the abnormality in this 7T image? Make textual response and locate the abnormality in this figure, returning the annotated image via downloading link.” 11 T1WI and 11 T2WI images from four patients were assessed in this task.

In Task 4, we further examined whether ChatGPT-4V is capable of integrating information from 7T images with imaging data from other modalities and give an accurate diagnosis. The 7T T1WI and T2WI data were combined with a PET image and a 3T contrast-enhanced T1 (T1-C) image as input. The data were from eight patients with brain tumors. The question prompt was written as “What is the disease depicted in the given images?”. We expected ChatGPT-4V to identify the respective abnormalities in each modality of the input and make a diagnosis of the possible disease. Additionally, we explored an image-based genetic diagnosis process using ChatGPT-4V and multimodality images. We asked ChatGPT-4V “Can you guess the IDH status from these images?”, and investigated whether ChatGPT-4V could infer the IDH genetic status from the given imaging manifestations.

2.5 Statistical analysis

For Tasks 1 and 2, we used accuracy to quantify the performance of ChatGPT-4V. For Tasks 3 and 4, a human-based evaluation was implemented to evaluate the textual answers, allowing the quantifications to involve more aspects than just accuracy. Three experienced experts were instructed to read through the images, questions, and corresponding answers from ChatGPT-4V and to rate the answers from 1 to 5. In Task 3, scores were calculated based on diagnostic accuracy, diagnostic basis, and logic smoothness. For the lesion localization test, if the given circle contained any lesion, the answer was regarded as accurate (score 5); otherwise, it was scored as zero. In Task 4, scores were calculated based on integrated diagnostic accuracy, diagnostic basis, logistic smoothness, multimodal integration ability, and readability.

3.1 Task 1-the basic conceptual tests for 7T images and ChatGPT-4V

First, we confirmed that ChatGPT-4V correctly encoded the knowledge about 7T MRI (Figure 2a). In its response, it correctly explained the concept of magnetic field strength (Tesla) and stated that the major differences between 3T and 7T are their image resolution, signal-to-noise ratio, and chemical sensitivity. In addition, ChatGPT-4V successfully differentiated 3T and 7T images based on its vision ability and its knowledge about 3T and 7T MRI data (Figure 2b). An example input image is the left-most one in Figure 2c). ChatGPT-4V provided satisfactory reasoning by highlighting that its judgment was based on image resolution and contrast. ChatGPT's ability to differentiate 3T and 7T images was reliably high, obtaining accuracy of 84.4% for 32 test images (eight groups of testing data). Partial data is shown in Figure 2c, supporting the idea that ChatGPT-4V has a preliminary concept basis for 7T images, and could be aware of their advantages when performing tasks with 7T images as inputs.

3.2 Task 2-modality recognition tests for 7T images using ChatGPT-4V

In Task 2, we further tested ChatGPT-4V's ability to recognize different 7T imaging modalities. In Figure 3, ChatGPT-4V successfully identified T1WI, T2WI, amide proton transfer-CEST, and ADC, and explained the characteristics of each of them. In addition, it can be observed that this accurate modality recognition was reproducible for another set of image inputs (see Supplementary File 2). Note that although it does not directly identify FA, MD, and colFA, it did identify that these three images were related to diffusion-weighted image (DWI), or suggested color-coded fiber tractography from DTI. Therefore, we scored these answers as 0.5 during the accuracy calculations. Overall, ChatGPT-4V scored 30 for 38 images (78.9% accuracy). ChatGPT-4V robustly identified T1WI, T2WI, and CEST, but tended to fail in differentiating between ADC, MD, and FA (see Supplementary File 2). In addition, we observed that ChatGPT-4V could not identify CEST and ignored one of the given images when there was no explicit hint about CEST in the prompt (see Supplementary File 2).

3.3 Task 3-single-modality-based diagnosis using 7T images and ChatGPT-4V

In Task 3, we tested the diagnostic capacity of ChatGPT-4V using single-modality 7T images. In this task, we required ChatGPT not only to generate an accurate diagnosis based on the imaging information but also to respond with smooth logics, readability for doctors and patients, and provide high interpretability for the diagnosis. Therefore, we asked ChatGPT to identify the abnormality/lesion in the given image and respond with a textual answer and annotations (Figure 4 a, b). Eleven cases were included in this task and three experts were invited to evaluate the answers. Overall, ChatGPT's performance was not satisfactory in this task, particularly failing in the lesion annotation (Figure 4c average score of 9.27/20), as agreed by all three clinical experts. In most of the tested cases (Supplementary Table 1), ChatGPT did not provide a precise diagnosis with a correct diagnostic basis observed from imaging manifestations. Only in three cases (Cases 2, 8, and 9) did the answers from ChatGPT score more than 3 points for both T1WI and T2WI. Additionally, there were only three cases where ChatGPT annotated the lesion accurately. These observations suggest that ChatGPT-4V is not capable of assisting in image-based diagnosis using single-modality 7T images, especially for small lesions. In addition, ChatGPT's decisions were not interpretable because it made decisions based on irrelevant manifestations in the given images. Interestingly, although based on limited results, it seems that the performance of ChatGPT-4V was better with T2WI, which could indicate a potential bias of ChatGPT-4V.

3.4 Task 4-multiple-modality-based diagnosis using 7T images and ChatGPT-4V

The capacity of ChatGPT-4V for multimodal-based diagnosis was explored using 7T MRI and PET images. In addition to conducting diagnosis for brain tumors, we attempted to instruct ChatGPT to infer the IDH mutant status (representative gene of molecular diagnosis) of the given image, which is vital information for clinical decision making and prognosis. Eight patients with different grade gliomas were included in this task. We demonstrate the setting of this task in Figure 5, with data from a patient with high-grade IDH wild-type disease (Figure 5a) and a patient with an IDH mutant WHO grade 2 astrocytoma (Figure 5b). In Figure 5a, we can observe that ChatGPT can recognize each presented modality, extract diagnosis-related information from each image, and make an accurate diagnosis about the tumor status. In addition, it comprehensively described the imaging genetics knowledge about multimodal imaging manifestations related to IDH status and provided a correct inference for IDH-wildtype. However, ChatGPT performed wrong analysis for low grade glioma (Figure 5b), resulting in an incorrect diagnosis of amyloid-related imaging abnormalities (ARIA, which can be seen in Alzheimer's disease or dementia). For the IDH status, it did not provide an explicit answer, stating that the presented images were not showing typical characteristics. This situation was reproduced for both IDH-mutant cases, and therefore could relate to a systematic safety setting of ChatGPT. Additionally, we note that ChatGPT understood the PET images based on default knowledge of fluorodeoxyglucose-PET (reflecting metabolism). This seems understandable because there is no clear clue in the presented image to indicate the exact type of PET, including the presented amino acid PET.

In quantitative analysis, ChatGPT achieved high scores for all dimensions in the task (Figure 5c, on average 21.17/25 scores), as rated by all three experts. A slight loss in score resulted from the dimension of integrated diagnosis, with ChatGPT making a completely wrong diagnosis for a Grade 2 glioma with IDH mutant status (Figure 5b) and a partially imprecise response for another low-degree IDH-mutant case (Supplementary File 4). These results support the capacity of ChatGPT to integrate multimodal imaging information and perform a diagnosis.