2.1 Description of the Study Population

This study utilized a subset of anonymized LDCT images sourced from NLST [30, 31] which has been used as a nested case–control sample by Cherezov et al. [8] in an earlier study. The NLST is a controlled, multicenter trial that compares the effectiveness of LDCT scans and chest X-rays to detect lung cancer in high-risk individuals. Participants in the trial were current or former smokers, aged 55 to 77, with a smoking history of at least 30 pack-years; and those who had quit had done so within the last 15 years. The NLST conducted three annual screenings: an initial baseline screen (T0) and two subsequent annual follow-up screens (T1 and T2). According to the trial's criteria, a positive LDCT screen was defined by the detection of one or more non-calcified nodules or masses exceeding 4 mm in axial diameter, or less frequently, other signs such as adenopathy or pleural effusion. Participants with positive screening results received follow-up instructions according to the trial's guidelines. The cohort selected for this research excluded participants with only negative screening results, which were defined as scans showing no abnormalities, minor abnormalities not suggestive of lung cancer, or significant abnormalities deemed not suspicious for lung cancer.

2.2 Nested Case–Control Study Framework

A schematic representation of the nested case–control study has been shown in Figure 1. In this study, we analyzed a total of 462 cases with positive initial screenings (T0). From this group, 160 participants were diagnosed with lung cancer during the first (T1) or second (T2) follow-up screenings, labeled as positive cases. And, 302 participants were diagnosed as benign during all three screenings, labeled as negative cases. Based on these frameworks, two distinct cohorts were established. Cohort 1 consisted of positive cases diagnosed with lung cancer at T1, and Cohort 2 included those diagnosed at T2. The remaining negative cases, who had positive initial screenings but were not diagnosed with lung cancer from T0 through T2, were distributed between the two cohorts following a 2:1 nested case–control study design in a way that the age, sex, race, and smoking status distribution matched between negative and positive cases in both cohorts. This resulted in minimizing the influence of confounders between two groups, as described in earlier studies [8, 32]. This stratification of samples in a nested case–control framework resulted in 168 negative cases in Cohort 1 and 133 in Cohort 2. Ultimately, Cohort 1 served as the training dataset with 254 samples, while Cohort 2 was utilized as the validation dataset with 208 samples. This approach allowed us to use the baseline CT images to train and validate lung cancer risk prediction models for 1- and 2-year intervals.

2.3 CT Annotation

Before the extraction of radiomic features from regions of interest, it was necessary to identify and segment lung nodules in both positively and negatively diagnosed cases in T0, T1, and T0 screening timelines. The nodule segmentations were obtained from the Schabath et al. study [32], which were performed manually by two radiologists, YL and QL, who reviewed all LDCT images. The process of identifying cancerous nodules among detected lung cancers during screenings utilized data from NLST, which provided details on nodule size and location. In instances where location data were incomplete, YL, the primary radiologist, manually identified and traced each nodule from T0 to T1. For participants with multiple nodules, the largest nodule identified at T0, and its changes through subsequent screenings were selected for the extraction of radiomic features [33].

2.4 Radiomic Feature Extraction and Preprocessing

Before the extraction of radiomic features, in-plane resampling was performed using the B-spline interpolation method from the SimpleITK (Simple Insightful Toolkit) library. This method adjusted the voxel size to (1,1, slice thickness) during preprocessing, which normalized the pixel spacing while maintaining the original slice thickness from the non-resampled images. Normalizing is crucial for ensuring consistency, standardized representation, and proper voxel alignment. The choice to preserve the original slice thickness was strategic, aiming to later harmonize the extracted radiomic features based on batch parameters, including variations in slice thickness. Also, the bin width of 0.03 was used for the gray-level discretization of the ROIs. Within the preprocessing step, the gray levels (voxel intensities) are resampled within the ROI to 2ⁿ number of bins, where n varies from 3 to 8 in the literature [36, 37]. This approach not only reduces noise but also increases the stability of the features [38, 39]. The bin width of 0.03 was determined based on the range of the first-order feature for the whole studies in this research, which was between 0.3 and 3 for the majority of the studies. This choice ensures that the number of bins remains between 10 and 100, a range that has demonstrated good reproducibility and performance in previous studies when using a fixed bin count [34, 40].

For the majority of studies, CT images were found for either T0 or T1 timelines with different reconstruction kernels used for image reconstruction. For each study, we selected CT images with the same reconstruction kernel at both T0 and T1 screenings. If multiple CT image pairs were found each with the same reconstruction kernel, one was chosen randomly, as there was no specific preference for the type of reconstruction kernel used.

2.5 Baseline and Delta Features

In this study, we analyzed six sets of radiomic features derived from CT scans taken at T0 and T1. For our analysis, we considered three main feature sets: the original features from T0, the delta features, and a combination of both, labeled as T0 + delta. Each of these sets was analyzed both before and after harmonization, resulting in six distinct sets of features used in this research. In addition, to explore changes in radiomic features over time, we also computed the difference between the features extracted at T0 and T1 time points, which is termed delta features in this work. This approach provided a comprehensive overview of the changes in nodules over time while controlling for potential confounding factors.

2.6 Harmonization of Features

In this study, we employed the Nested ComBat [19] method for harmonizing radiomic features based on two batch parameters, that is, slice thickness category (TC) and reconstruction kernel type (KT), independently. This step was adjusted for variations due to slice thickness and reconstruction kernels, ensuring standardized comparisons across all scans. Radiomic features extracted from CT images taken at T0, T1, and T2 were stratified into three slice thickness categories: 1.0 to 2.0 mm, 2.0 to 3.0 mm, and 3.0 to 4.0 mm. The reconstruction kernels used were also categorized into sharp and soft types. Sharp kernels included types such as “D,” “B70f,” “B80f,” “BONE,” “FC82,” “LUNG,” “B60f,” “FC51,” “FC30,” “FC50,” while soft kernels were identified as “B30f,” “B30s,” “FC10,” “FC02,” “FC01,” “A,” “STANDARD,” “B,” “C,” “B50s,” and “B50f.” Given that the CT images were produced by four different manufacturers, and considering that kernel implementation scan vary across these manufacturers, we aligned the kernels based on existing research that explores how reconstruction kernels influence radiomic features [41]. The clinical covariates used along with TC and KT, serving as batch effects, were smoking, gender, and age.

The Nested ComBat technique expands upon the traditional ComBat approach, which typically addresses a single batch effect, by accommodating multiple concurrent influences. Nested ComBat initiates the harmonization process with a list of predetermined batch effects and clinical covariates alongside the initial dataset of radiomic features. The process is iterative, that is, the features are adjusted individually for each batch effect in each cycle. This step is crucial for maintaining the clinical relevance of the features while ensuring that prior adjustments for other batch effects are not retained, thereby fostering a more focused and effective correction. After adjusting the features in each cycle, the Kolmogorov–Smirnov (KS) test is applied to identify any significant discrepancies in the distributions attributable to each batch effect. The feature set that shows the minimal distributional disparities is then chosen for further harmonization, and the associated batch effect is removed from the list for subsequent iterations. This process continues until all identified batch effects have been systematically addressed. The final harmonized feature set is rigorously analyzed to confirm that it exhibits no significant distribution differences when stratified by the original batch effects [19]. This meticulous approach to harmonization is vital to ensure the accuracy and reliability of our analysis, particularly in assessing the temporal changes in nodules across CT scans. We also performed the principal component analysis (PCA) to assess the impact of the Nested ComBat harmonization process on radiomic features.

2.7 Machine Learning Models

In this study, we employed nine ML models to evaluate the predictive power of six sets of original and harmonized radiomic features for lung cancer risk prediction. The models used were Random Forest (RF), Decision Tree (DT), Gaussian Naive Bayes (GNB), Support Vector Classifier (SVC), AdaBoost (Ada), XGBoost (XGB), Gradient Boosting (GB), Logistic Regression (LR), and K-Nearest Neighbors (KNN). These models were chosen for their diverse approaches to handling classification tasks, allowing us to comprehensively assess the effectiveness of the feature sets in predicting lung cancer risk. These classification-based models were implemented using the scikit-learn package in Python, a library that easily facilitates the application of numerous ML algorithms. This streamlined the integration of various predictive modeling techniques into our analysis pipeline.

2.8 Feature Reduction and Selection Strategies

2.9 Balancing the Dataset

To address the challenge of our unbalanced 2:1 nested case–control dataset and to achieve a balanced dataset, we employed the Synthetic Minority Oversampling Technique (SMOTE) [47] implemented in Imbalanced-learn, an open-source MIT-licensed library, relying on scikit-learn package in Python. This oversampling method is specifically designed to enhance the representation of minority classes in datasets that are not balanced. Unlike simple random oversampling that duplicates existing samples, SMOTE generates synthetic samples by interpolating between several minority class instances. This technique not only helps in creating a more balanced dataset but also potentially improves the performance of ML models by providing a richer and more diverse set of training examples. In total, we utilized SMOTE in 135 models for each category of features: T0, delta, and T0 + delta. These models were generated through combinations of nine ML models and five feature selection methods, applied to three types of non-harmonized, TC-based, and KT-based features. The results were compared to those obtained using the original unbalanced dataset.

2.10 Analysis Workflow

The pipeline of the radiomic-based predictive models in our study presented in Figure 2 included three main steps: preprocessing, pre-validation (training), and validation.

3.1 Study Population Characteristics

Figure 3 outlines the demographic details of the training and validation cohorts used in this study, categorized by positive and negative cases.

The cohorts were established in such a way that the negative cases matched the positive cases in terms of age, sex, race, and smoking status, ensuring minimal differences in the distribution of these characteristics across the training and validation cohorts as well as between the positive and negative cases. Across the cohort of 462 samples, the individuals have an average age of 63.6 ± 5.0 years. Specifically, the training cohort consists of 254 individuals with an average age of 64.0 ± 5.2 years, while the validation cohort includes 208 individuals, averaging an age of 62.9 ± 4.8 years. Regarding smoking status, 51% of the patients in the training cohort and 54% in the validation cohort are smokers, compared to 49% and 46% who are non-smokers, respectively. To validate the matching process, statistical tests were conducted to assess differences between positive and negative cases in age, sex, race, and smoking status. The t-test for age revealed a p-value of 0.72, indicating no significant difference between groups. The chi-square test for sex produced a p-value of 0.41, suggesting no significant association. For race, the chi-square test yielded a p-value of 0.06, indicating a trend toward significance. Finally, the chi-square test for smoking status resulted in a p-value of 0.50, showing no significant difference. These results confirm that the cohorts were successfully matched, with minimal differences in demographic and clinical characteristics.

3.2 Radiomic Feature Extraction

Using the Pyradiomics pipeline, out of 1231 radiomic features, 113 features categorized into shape-based (27 features), intensity-based (18 features), and texture-based (68 features) were extracted directly from the original CT images, while 1118 intensity and texture-based features were extracted from derived images that were obtained using various filters: 688 features using a Wavelet filter, and 86 features each from LoG, Gradient, Square, Square-root, and Exponential filters. Radiomic features were extracted from 1694 CT images available at T0, T1, and T2. Not all images were included in the training and validation cohorts. For instance, some images were produced using two different reconstruction kernels for the same patient. Furthermore, features extracted from T2 were also not utilized in ML analyses. Nevertheless, to enhance the robustness of our data for effective harmonization, we extracted radiomic features from as many CT scans as possible. This approach aimed to enrich our dataset with a sufficient number of samples from each TC and KT, enhancing the accuracy and effectiveness of the harmonization process for more precise feature adjustment.

3.2.1 Harmonization of Radiomic Features

Radiomic features were extracted from 1689 out of the available 1694 CT images available, with five images excluded due to missing clinical data, slice thickness, or reconstruction kernel information. The slice thicknesses across these images ranged from 1.0 to 3.2 mm, with a median of 2.5 mm. For harmonization purposes, the images were stratified into three categories based on slice thickness: 1.0–2.0 mm, 2.0–3.0 mm, and 3.0 to 4.0 mm, resulting in distributions of 112, 1410, and 164 samples, respectively. Additionally, images were classified by reconstruction kernels into sharp (389 images) and soft (1300 images). After the harmonization of the radiomic features based on the TC and KT independently, and to better understand the impact of Nested ComBat harmonization on these features, we utilized principal component analysis (PCA). This approach facilitates the projection and visualization of high-dimensional data. Figure 4A,B illustrates the distribution of radiomic features based on PCA results, before and after harmonization. The left panel of Figure 4A highlights the groups by TC while the right panel presents the distinct groups based on KT before harmonization. As can be seen in both panels of Figure 4A, before harmonization clustering by TC or KT is observable, though the clusters are not significantly distinct. The left panel in Figure 4B presents the distribution of the features after harmonization based on TC while the right panel presents it when harmonization is done based on KT. Both panels highlight how the harmonization process which was done on TC and KT independently, effectively reduces discrepancies caused by different TC or KT groups, leading to a more homogeneous dataset with reduced variabilities between the TC or KT clusters.

3.3 Performance of Models

In total, we developed 270 models for each category of extracted radiomic features- T0, delta, and T0 + delta- by integrating nine ML models and five feature selection methods which were carried out on non-harmonized, TC- and KT-based harmonized features with and without SMOTE. The maximum number of features used in developing ML models, determined through the LASSO feature reduction technique, ranged from 14 to 46. This variability was dependent on the feature category and the type of harmonization applied. The importance of features selected by LASSO along with LASSO curves for each of these categories has been plotted in Supplementary file, Figures S1–S9. The AUC ranges varied based on the use of SMOTE, harmonization, and type of harmonization. The median performance of each of the nine ML models over all five feature selection methods using T0, delta, and T0 + delta feature sets has been presented in Figure S10A–C in the supplementary file.

For the T0 features, the range of AUC on the validation dataset without SMOTE was [0.56, 0.72] for non-harmonized features, [0.54, 0.69] for KT-based harmonized features, and [0.51, 0.69] for TC-based harmonized features. With the application of SMOTE, the corresponding ranges were [0.60, 0.68], [0.54, 0.69], and [0.59, 0.69]. The highest AUC observed was 0.72 on the validation dataset, paired with a training score of 0.73 ± 0.07, achieved by the GB model using non-harmonized features and the Surf feature selection method without SMOTE. On the other hand, the lowest AUC on the validation dataset was 0.51, with a training dataset score of 0.72 ± 0.04, obtained using an SVC with the F-test feature selection method on TC-based harmonized features without SMOTE. The application of SMOTE improved the minimum AUC when each of the non-harmonized, KT-based, and TC-based harmonized features were employed; however, the maximum AUC did not improve by SMOTE.

Regarding the delta features, AUCs on the validation set without SMOTE ranged from [0.60, 0.73] for non-harmonized features, [0.64, 0.72] for KT-based, and [0.60, 0.74] for TC-based harmonized features. When SMOTE was implemented, these AUCs altered to [0.61, 0.76], [0.62, 0.74], and [0.64, 0.76] respectively. The maximum AUC achieved was 0.76 on the validation set, corresponding to a training score of 0.85 ± 0.03, found in the GB model with TC-based harmonized features and the MultiSurf selection method with SMOTE. The combination of the RF model with the MI feature selection method and SVC with the Surf feature selection method also resulted in the maximum AUC of 0.76 on the validation dataset using non-harmonized features with SMOTE. The minimum AUC of 0.60 on the validation dataset was noted in the DT model employing the F-test feature selection method with non-harmonized features without SMOTE, achieving a training performance of 0.81 ± 0.03. Similarly, a DT model utilizing the MultiSurf feature selection method with TC-based harmonized features, also without SMOTE, recorded the same AUC on validation, with a training performance of 0.84 ± 0.06.

For the combined T0 + delta features, the analysis of AUCs on the validation dataset without the use of SMOTE showed a range of [0.64, 0.73] for non-harmonized features, [0.63, 0.74] for KT-based harmonized features, and [0.62, 0.73] for TC-based harmonized features. Upon applying SMOTE, these AUCs shifted to ranges of [0.58, 0.74], [0.60, 0.76], and [0.56, 0.76]. The highest AUC achieved on the validation dataset was 0.76, obtained in two scenarios: with an SVC model using KT-based harmonized features and the F-test feature selection method with SMOTE, and with an XGB model applying the MI feature selection method on TC-based harmonized features with SMOTE, each yielding a training AUC of 0.86 ± 0.02. In contrast, the minimum AUC on the validation dataset was observed as 0.56, associated with a training score of 0.83 ± 0.05, derived from a DT model that utilized TC-harmonized features and the F-test feature selection method with SMOTE. This indicates variability in model performance based on feature selection and harmonization approaches, underlining the importance of choosing the right combination for optimal prediction.

Figure 5 presents a comparison of the best-performing models on the validation cohort for each feature set (T0, delta, and T0 + delta), considering both scenarios with and without Nested ComBat harmonization based on KT and TC. The results highlight that utilizing SMOTE generally enhances outcomes; specifically, harmonization based on TC improves results across all feature sets when SMOTE is applied, while harmonization based on KT does not enhance performance for the delta features. Notably, the highest-performing model, which uses SMOTE with delta features and TC-based harmonization, achieved an AUC score of 0.76. To further support these observations, ANOVA was conducted to test for significant differences between AUCs of models utilizing SMOTE and not utilizing SMOTE across the different feature sets. The results indicate that there is no significant difference between them for the T0-based models (p = 0.335). For the delta-based models, a p-value of 0.062 suggests a close-to-significant difference. Moreover, in the case of the models using T0 + delta features, a significant difference was found (p = 0.046), indicating that the combination of T0 and delta features significantly affects model performance when using SMOTE.

Additionally, Figure 6 extends this comparison across different feature sets. The results demonstrate that models using delta features consistently outperform those using only T0 features, regardless of the SMOTE application. This is further supported by a statistically significant difference between the AUC scores for models using T0 and delta features, with an ANOVA test yielding a p-value of less than 0.0001, confirming that the difference is highly significant. However, the addition of delta features to T0 does not yield improvements over using delta features alone, as indicated by an ANOVA test comparing delta and T0 + delta models, which produced a non-significant p-value of 0.823. Similarly, a significant difference is observed between models using only T0 features and those using T0 + delta features (p < 0.0001), suggesting that while adding delta features enhances performance compared to T0 alone, it does not outperform delta features used independently.