Original Article

Open Access

Utility of Machine Learning Algorithms in Predicting Preoperative Lymph Node Metastasis in Patients With Rectal Cancer Based on Three-Dimensional Endorectal Ultrasound and Clinical and Laboratory Data

Weiqin Huang MM

Department of Ultrasonography, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, China

Search for more papers by this author

Ruoxuan Lin BS,

Ruoxuan Lin BS

Department of Ultrasonography, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, China

Search for more papers by this author

Xiaohui Ke BS,

Xiaohui Ke BS

Department of Ultrasonography, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, China

Search for more papers by this author

Shixiong Ni BS,

Shixiong Ni BS

Department of Ultrasonography, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, China

Search for more papers by this author

Zhen Zhang MD,

Corresponding Author

Zhen Zhang MD

Department of Ultrasound, First Affiliated Hospital of China Medical University, Shenyang, China

Address correspondence to Lina Tang, Department of Ultrasonography, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, No. 420, Fuma Road, Jin'an District, Fuzhou 350014, China.

E-mail: [email protected]

Zhen Zhang, Department of Ultrasound, First Affiliated Hospital of China Medical University, No. 155, Nanjing Street, Heping District, Shenyang, China.

E-mail: [email protected]; [email protected]

Search for more papers by this author

Lina Tang BS,

Corresponding Author

Lina Tang BS

[email protected]

orcid.org/0000-0001-6945-4973

Department of Ultrasonography, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, China

E-mail: [email protected]

Zhen Zhang, Department of Ultrasound, First Affiliated Hospital of China Medical University, No. 155, Nanjing Street, Heping District, Shenyang, China.

E-mail: [email protected]; [email protected]

Search for more papers by this author

Weiqin Huang MM,

Weiqin Huang MM

Department of Ultrasonography, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, China

Search for more papers by this author

Ruoxuan Lin BS,

Ruoxuan Lin BS

Department of Ultrasonography, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, China

Search for more papers by this author

Xiaohui Ke BS,

Xiaohui Ke BS

Department of Ultrasonography, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, China

Search for more papers by this author

Shixiong Ni BS,

Shixiong Ni BS

Department of Ultrasonography, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, China

Search for more papers by this author

Zhen Zhang MD,

Corresponding Author

Zhen Zhang MD

Department of Ultrasound, First Affiliated Hospital of China Medical University, Shenyang, China

E-mail: [email protected]

Zhen Zhang, Department of Ultrasound, First Affiliated Hospital of China Medical University, No. 155, Nanjing Street, Heping District, Shenyang, China.

E-mail: [email protected]; [email protected]

Search for more papers by this author

Lina Tang BS,

Corresponding Author

Lina Tang BS

[email protected]

orcid.org/0000-0001-6945-4973

Department of Ultrasonography, Clinical Oncology School of Fujian Medical University, Fujian Cancer Hospital, Fuzhou, China

E-mail: [email protected]

Zhen Zhang, Department of Ultrasound, First Affiliated Hospital of China Medical University, No. 155, Nanjing Street, Heping District, Shenyang, China.

E-mail: [email protected]; [email protected]

Search for more papers by this author

First published: 04 July 2023

https://doi.org/10.1002/jum.16297

This work was supported by Joint Funds for the Innovation of Science and Technology, Fujian Province (grant number: 2021Y9220) and the Fujian Provincial Health Commission (grant number: 2018-ZQN-14).

The authors declare that they have no competing interests.

We would like to acknowledge the hard and dedicated work of all the staff that implemented the intervention and evaluation components of the study.

Share a link

Email
Wechat
Bluesky

Abstract

Background

We aimed to investigate the value of a machine learning (ML) algorithm in the preoperative prediction of lymph node metastasis in patients with rectal cancer.

Methods

Based on the histopathological results, 126 rectal cancer patients were divided into two groups: lymph node metastasis-positive and metastasis-negative groups. We collected clinical and laboratory data, three-dimensional endorectal ultrasound (3D-ERUS) findings, and parameters of the tumor for between-group comparisons. We constructed a clinical prediction model based on the ML algorithm, which demonstrated the best diagnostic performance. Finally, we analyzed the diagnostic results and processes of the ML model.

Results

Between the two groups, there were significant differences in serum carcinoembryonic antigen (CEA) levels, tumor length, tumor breadth, circumferential extent of the tumor, resistance index (RI), and ultrasound T-stage (P < 0.05). The extreme gradient boosting (XGBoost) model had the best comprehensive diagnostic performance for predicting lymph node metastasis in patients with rectal cancer. Compared with experienced radiologists, the XGBoost model showed significantly higher diagnostic value in predicting lymph node metastasis; the area under curve (AUC) value of the receiver operating characteristic (ROC) curve of the XGBoost model and experienced radiologists was 0.82 and 0.60, respectively.

Conclusions

Preoperative predictive utility in lymph node metastasis was demonstrated by the XGBoost model based on the 3D-ERUS finding and related clinical information. This could be useful in guiding clinical decisions on the selection of different treatment strategies.

Abbreviations

3D-ERUS: three-dimensional endorectal ultrasound
AUC: the area under curve
CEA: carcinoembryonic antigen
CT: computed tomography
ERUS: endorectal ultrasonography
ML: machine learning
MRI: magnetic resonance imaging
RC: rectal cancer
RF: random forest
RI: resistance index
ROC: receiver operating characteristic
SVM: support vector machine
TME: total mesorectal excision
XGBoost: extreme gradient boosting

Rectal cancer (RC) is one of the most common malignant tumors of the digestive tract, affecting 700,000 new patients worldwide annually.¹ According to the latest data from GLOBOCAN 2021,² RC ranks eighth among all cancers worldwide in terms of both incidence (3.9%) and mortality (3.2%). Although total mesorectal excision (TME) is the standard surgical treatment of mid and low RC,³ positive lymph nodes suggest a risk of systemic metastasis and an unfavorable prognosis.⁴ Thus, patients with locally advanced RC (tumor categories 3–4 or node categories 1–2) require multimodal therapy to reduce the rate of local recurrence.⁵ Therefore, accurate preoperative evaluation of lymph node metastasis is important for facilitating treatment decisions and evaluating prognoses.

Preoperative evaluation of lymph nodes is done using imaging investigations such as computed tomography (CT), magnetic resonance imaging (MRI), and endorectal ultrasonography (ERUS),^6-8 whereas ERUS and CT showed similar accuracy. The sensitivity and specificity of CT and ERUS were 79 and 76%, and 58 and 80%, respectively.⁹ Therefore, meta-analyses and population data demonstrate that clinical nodal staging, even with the combination of ERUS, CT, and MRI, is unreliable.¹⁰

Machine learning (ML) algorithms have been widely used in the medical field for outcome prediction, diagnosis, treatment management, and prognosis.^{11, 12} However, the logic and complexity of various ML algorithms vary,¹³ and there may also be differences in their clinical applications. A previous study¹⁴ compared the diagnostic performance of different ML algorithms, such as the support vector machine (SVM), k-nearest neighbor, random forest (RF), extreme gradient boosting (XGBoost), and light gradient-boosting machine (LightGBM), in the diagnosis of benign and malignant breast lesions. They found that the LightGBM model showed the best among the five ML models, achieving 99.86% accuracy, 100.0% precision, 99.60% recall, and 99.80% on the F1-score. Further, studies have demonstrated the potential of an ML model to preoperatively predict lymph node metastases in patients with RC. However, ML remains limited to MRI and CT images.^{15, 16} Consequently, in the present study, we developed and validated an ML algorithm for the preoperative prediction of lymph node metastasis of RC based on the three-dimensional endorectal ultrasound (3D-ERUS) findings, along with related clinical information, carcinoembryonic antigen (CEA), and carbohydrate antigen 19-9 (CA199) values.

Materials and Methods

Study Participants

Medical records of patients diagnosed with rectal cancer and admitted to our hospital between May 2020 and September 2022 were analyzed retrospectively. The inclusion criteria were the following: 1) Patients with rectal cancer confirmed pathologically by preoperative enteroscopy biopsy; 2) Patients whose preoperative blood test findings for CEA and CA199 were available; and 3) Patients whose 3D-ERUS findings within 1 week before surgery were available. All patients underwent total mesorectal excision and did not undergo neoadjuvant chemoradiotherapy. This study was approved by the Ethics Committee of the Fujian Cancer Hospital (2017-044-01, K2022-168-01), which waived the requirement of informed consent given the retrospective nature of this study.

Instruments

Ultrasonography was performed using a Flex Focus ultrasound scanner (BK Company, Denmark) with a rectal 3D probe frequency of 8–15 Hz. Intestinal cleaning was performed using a cleansing enema or two prescriptions of Hengkangzhengqing produced by Jiangxi Hengkang Pharmaceutical Co, Ltd, a laxative approved for bowel clearing, in our hospital and administered as a 2000 mL suspension before the examination.

Inspection Protocol

Intestinal cleaning was performed 4–6 hours before the imaging procedure. During 3D-ERUS, the patient was asked to lie on the left side with the hip and knee bent. An ultrasound examination was performed after injecting 60 mL of couplant with an enema irrigator into the rectum through the anus. The surface of the ultrasound probe was coated with latex and coated with coupling. Subsequently, the probe was slowly inserted into the rectum beyond the injury site. The depth of the probe was adjusted to allow the detection of the main part of the lesion. Next, when the tumor was detected, the tumor location, size, shape, internal echo, blood flow, depth, and extent of tumor infiltration of the intestinal wall were observed and recorded. Additionally, we determined whether there was tumor infiltration of surrounding tissues and organs, the number of perirectal lymph nodes, and the dimensions of the largest lymph node. Finally, two physicians with over 5 years' experience in ultrasound diagnosis analyzed the ultrasound images and determined the ultrasound T and N staging.

Evaluation Standards

We did the ultrasound staging of tumor infiltration depth (uT staging) as per the staging criteria of Beynon et al^{17, 18} as follows: 1) uTl stage: Tumor location is limited to the mucosa and submucosa and ultrasound shows integration of the second layer of the hyperechoic zone from the inner intestinal wall to the outer membrane; 2) uT2 stage: There is evidence of infiltration of tumor to the intrinsic muscular layer, the ultrasound shows destruction of the second layer of the hyperechoic zone in the local lesion, and thickening of the outer muscular layer (second layer of the hypoechoic zone), and there is integration of the second layer of the hyperechoic zone; 3) uT3 stage: The tumor involves the whole layer and there is evidence of infiltration of the fibrous perirectal fibrous adipose tissue and the ultrasound shows destruction of the third layer of the hyperechoic zone of the local lesion and irregular serrated protuberance of the hypoechoic zone involving the peri-intestinal tissues; and 4) uT4 stage: The tumor is eroding adjacent organs or tissues (including the prostate and vagina), with ultrasound showing disappearance of the hyperechoic zone in the normal margin of organs surrounding the lesion sites, as well as no boundary between the hyperechoic and hypoechoic zones.

Machine Learning Algorithms

Data Pre-Processing

Since TME surgery was performed in all cases in this article, all patients were inpatients. The clinical data (gender, age, BMI [weight (kg)/height (cm)]) of the patients were collected immediately after the patients arrived at the hospital. The laboratory examination (CA199, CEA) was the first fasting blood test for the patients conducted in the hospital. The 3D ultrasound data (distance from the lower edge of the lesion to the anal edge, distance from the lower edge of the lesion to the inner edge of the upper edge, long diameter of the lesion, short diameter of the lesion, RI value in the lesion) can be directly presented by 3D ultrasound images. The scope of the lesion to the perianal area % can be interpreted by doctors through ultrasound images. The T stage of the lesion was diagnosed by doctors through images and personal clinical experience.

Data analyzed in this study included continuous variables such as age, body mass index (BMI), serum CA199 levels, serum CEA levels, the distance between the tumor and anal margin, distance from the lower tumor margin to the upper internal anal sphincter (IAS), tumor length, tumor breadth, V_max, and resistance index (RI), as well as categorical variables such as gender (male and female), circumferential extent of the tumor (25, 50, and 100%), and ultrasound staging (T1, T2, T3, and T4). Categorical variables were processed using an on-demand code.

Development of the Training and Test Sets

The 126 samples were randomly divided into training sets and test sets, following a ratio of 7:3. Samples were balanced to allow consistency of proportions between the train and test sets with that of the total data set.

We used the training set to train the model. Ten-fold cross-validation was performed on four ML algorithms, namely, the support vector machine (SVM), extreme gradient boosting (XGBoost), Random Forest (RF), and the Light Gradient Boosting Machine (LightGBM). The grid search method was used to optimize the parameters, which adjusted and optimized the diagnostic performance of all ML algorithms and avoided overfitting the model, improving the accuracy of model classification. We comprehensively evaluated the diagnostic performance of the four algorithms for predicting lymph node metastasis in patients with rectal cancer using the average AUC and average accuracy derived from the 10-fold cross-validation. The entire training data set was used for training the four ML models.

The diagnostic performance of the ML model was verified through the test set data, and the diagnostic performance of all models was evaluated using the ROC curve. We verified the fit of the best-performing model by analyzing the accuracy of the test. Then, we compared the best-performing ML algorithm with an experienced radiologist. There were two ultrasound radiologists; both are senior physicians with 5 years of clinical experience in colorectal cancer diagnosis. Finally, we evaluated the prediction utility of each variable for lymph node metastasis based on its importance identified during the algorithm application process.

Statistical Analyses

All statistical analyses were performed using SPSS (v. 26.0; IBM Corp., Armonk, NY) statistical software. Statistical significance was set at P < .05. K statistics were performed on 3D-EURS findings of lesions. Categorical variables are presented as number (percentage), with comparisons between groups done using the chi-squared test. Normally distributed continuous variables are presented as mean ± standard (x̄ ± s) deviation, with between-group comparisons done using the t-test. We used the rank sum test for between-group comparisons of non-normally distributed continuous variables. For data analysis, we used the four ML algorithms from the Scikit-learn (https://scikit-learn.org/stable/) packages in Python (version 3.8). Receiver operating characteristic (ROC) curves were drawn by combining the seaborn and matplotlib libraries and used to calculate the corresponding curve area, accuracy, sensitivity, specificity, negative predictive value, and positive predictive value. The classification efficiency of the model was evaluated using a confusion matrix.

Results

Clinical and Pathological Data of Patients

Among the 1803 patients with rectal cancer admitted to our hospital between May 2020 and September 2022, 599 patients underwent total mesorectal excision, 283 underwent neoadjuvant therapy before the operation, 148 did not undergo the 3D-ERUS examination, and 42 had imperfect 3D-ERUS images. In 42 patients, a) the tumor size could not be measured because the tumor was too large and not completely included in the 3D image collection; or b) in the acquisition of images, there was no color Doppler measurement; or c) the location of the distance from the anal margin was not completely collected, and the location of the tumor could not be defined. As a result, the data were incomplete and could not be selected for the study. Therefore, only the remaining 126 patients with rectal cancer (69 men and 57 women, age range: 27–78 years; mean age: 61.25 ± 10.36 years) who underwent preoperative staging with 3D-ERUS followed by surgery without neoadjuvant therapy were included in this study (Supplemental Figure 1).

In the two evaluations of the 3D-EURS findings of lesions, the intra-observer K value for radiologist A was 0.878–0.926, while the inter-observer K value for radiologists A and B was 0.866–0.906. Therefore, the evaluation of the 3D-EURS finding was reproducible and stable. All 3D-EURS findings of the lesions in this study were assessed by radiologist A.

We included 126 patients who were pathologically diagnosed with adenocarcinoma. Patients were divided into positive and negative lymph node metastasis groups according to the TNM staging of pathological results. As shown in Table 1, there were significant between-group differences in serum CEA levels, tumor length, tumor breadth, circumferential extent of the tumor, RI, and ultrasound stage T (P < .05), but not in age, gender, BMI, CA199 test, tumor location, and V_max value (Table 1).

Table 1. Comparison Between Patients With Positive and Negative Lymph Node Metastases

	Positive (47 Patients)	Negative (79 Patients)	Statistics (χ²/T/Z)	P-Value
Clinical information
Age	59.49 ± 9.60	62.30 ± 10.70	1.482	.141
			0.009	.923
Male	26 (55.32)	43 (54.43)
Female	21 (44.68)	36 (45.57)
BMI	23.56 ± 2.70	22.89 ± 3.37	1.168	.245
CA199			0.298	.766
CEA			2.275	.023
Lesion site
Distance to the anal margin (cm)	6.16 ± 2.09	5.55 ± 1.89	1.685	.095
Distance to the upper margin of the internal anal sphincter (cm)	2.25 ± 1.49	2.15 ± 1.52	0.359	.720
The long tumor diameter	3.62 ± 1.00	3.19 ± 1.21	2.017	.046
The short tumor diameter			2.278	.023
Tumor invasion circumference			8.687	.034
≤25% circumference	16 (34.04)	45 (56.96)
>25% and ≤50% circumference	24 (51.06)	30 (37.97)
>50% circumference and <100% circumference	4 (8.51)	1 (1.27)
100% circumference	3 (6.12)	3 (6.38)
CDFI
V_max			0.515	.607
RI	0.68 ± 0.09	0.64 ± 0.08	2.307	.023
Ultrasound stage T			12.117	.007
T1	4 (8.51)	13 (16.46)
T2	11 (23.40)	34 (43.04)
T3	27 (57.44)	31 (39.24)
T4	5 (10.64)	1 (1.26)

Note: V_max was used to measure the peak value of systolic blood flow in tumor arteries. CA199, CEA, tumor short diameter, and V_max values showed a non-normal distribution.
CDFI, color Doppler flow imaging; RI, blood flow resistance index of the tumor.

Compared with pathological examination, the precision of ERUS in detecting lymph node metastasis was 65.87% (83/126 cases). Among the 47 (37.30%) patients with lymph node metastases on pathological examination, 20 (20/47 cases, 42.55%) and 27 (27/47 cases, 57.46%) patients either had or lacked evidence of lymph node metastases on ERUS, respectively. Among the 79 (62.70%) patients without lymph node metastases on pathological examination, 63 (63/79, 79.75%) and 16 (16/79, 20.25%) patients either lacked or had positive findings of lymph node metastases on ERUS, respectively.

Choosing a Machine Learning Model

In the total data set (n = 126), 47 and 79 patients had positive and negative lymph node metastases, respectively (ratio, 1:1.68). In the training set (n = 88), there were 33 and 55 patients with positive and negative lymph node metastases, respectively (ratio, 1:1.67). In the test set (n = 38), 14 and 24 patients had positive and negative lymph node metastasis, respectively (ratio, 1:1.71). Consequently, the proportions between the training and test sets groups were consistent with those of the total data set (Table 2).

Table 2. Comparison Between the Train Set and the Test Set

	Train Set				Test Set
	Positive (33 Patients)	Negative (55 Patients)	Statistics (χ²/T/Z)	P-Value	Positive (14 Patients)	Negative (24 Patients)	Statistics (χ²/T/Z)	P-value
Clinical information
Age	59.40 ± 10.14	63.15 ± 10.58	1.636	0.106	59.71 ± 8.57	60.38 ± 10.95	0.194	.848
			0.196	0.658			0.009	.923
Male	17 (51.5)	31 (56.4)			9 (64.3)	12 (50.0)
Female	16 (48.5)	24 (43.6)			5 (35.7)	12 (50.0)
BMI	23.00 ± 2.22	22.94 ± 3.39	0.099	0.921	24.87 ± 3.31	22.77 ± 3.38	1.877	.070
CA199			0.388	0.698			0.061	.952
CEA			2.354	0.019			0.515	.607
Lesion site
Distance to the anal margin (cm)	5.73 ± 2.11	5.78 ± 2.02	0.103	0.918	7.17 ± 1.70	5.03 ± 1.46	4.105	.000
Distance to the upper margin of the internal anal sphincter (cm)	1.85 ± 1.23	2.18 ± 1.59	1.039	0.302	3.19 ± 1.67	2.06 ± 1.38	2.244	.031
The long tumor diameter	3.59 ± 1.00	3.20 ± 0.98	1.743	0.086	3.68 ± 1.02	3.16 ± 1.65	1.056	.298
The short tumor diameter			1.677	0.094			1.503	.133
Tumor invasion circumference			11.411	0.010			0.756	.450
≤25% circumference	12 (36.4)	34 (61.8)			4 (28.6)	11 (45.8)
>25% and ≤50% circumference	16 (48.5)	21 (38.2)			8 (57.1)	9 (37.5)
>50% circumference and <100% circumference	4 (12.1)	0 (0.0)			0 (0.0)	1 (4.2)
100% circumference	1 (3.0)	0 (0.0)			2 (14.3)	3 (12.5)
CDFI
V_max			0.241	0.809			0.575	.565
RI	0.67 ± 0.09	0.64 ± 0.08	1.532	0.129	0.70 ± 0.08	0.65 ± 0.09	1.884	.068
Ultrasound stage T			9.053	0.029			4.571	.206
T1	3 (9.1)	6 (10.9)			1 (7.1)	13 (29.2)
T2	9 (27.3)	28 (50.9)			2 (14.3)	34 (25.0)
T3	18 (54.5)	21 (38.2)			9 (64.3)	31 (41.7)
T4	3 (9.1)	0 (0.0)			2 (14.3)	1 (4.2)

Note: V_max was used to measure the peak value of systolic blood flow in tumor arteries. CA199, CEA, tumor short diameter, and V_max values showed a non-normal distribution.
CDFI, color Doppler flow imaging; RI, blood flow resistance index of the tumor.

Based on the 3D-ERUS finding and related clinical information (including the value of CEA and CA199, gender, age, and BMI), the four ML models of RF, XGBoost, SVM, and LightBGM were compared using Average-AUC and Average-Accuracy derived from the 10-fold cross-validation (Supplemental Table 1). It showed that the XGBoost model had the best Average-AUC (0.945) and Average-accuracy (0.883).

We used the entire data from the training set for training the four ML models. Then the diagnosis performance of the training set of the ML algorithms was compared using the value of accuracy, sensitivity, specificity, and the F1-score (Table 3).

Table 3. Evaluation of the Performance of Machine Learning Models (Train Set)

Classifier	AUC	Accuracy	Sensitivity	Specificity	F1-Score
RF	0.9970	0.9659	0.9091	1.0000	0.9655
SVM	0.9994	0.9886	1.0000	0.9818	0.9887
XGboost	0.9242	0.8863	0.8485	0.9091	0.8864
LightGBM	0.7579	0.6704	0.4242	0.8182	0.6569

External validation of the model-used test set showed that the XGBoost model performed better than the other ML models. As shown, the AUC of the model was 0.82, the accuracy was 84.21%, the sensitivity was 85.71%, the specificity was 83.33%, and the F1-score was 0.8439 (Table 4). Comparing the performance of both the training set and test set among all the ML algorithms, the XGBoost model showed a good fit (Supplemental Figures 2 and 3). Finally, we selected the XGBoost algorithm with the best diagnostic performance and compared the diagnosis performance with radiologist A.

Table 4. Evaluation of the Performance of Machine Learning Models (Test Set).

Classifier	AUC	Accuracy	Sensitivity	Specificity	F1-Score
RF	0.6681	0.7105	0.4286	0.8750	0.6927
SVM	0.6057	0.6579	0.4286	0.7917	0.6474
XGboost	0.8214	0.8421	0.8571	0.8333	0.8439
LightGBM	0.6518	0.5789	0.2857	0.7500	0.5601

We compared the training and test sets of the XGBboost model in terms of pathological results. In the training set, the prediction precision of positive and negative lymph node metastasis was 84.85% (28/33 cases) and 90.91% (50/55 cases), respectively. In the test set, the prediction precision of positive and negative lymph node metastasis was 85.71% (12/14 cases) and 83.33% (20/24 cases), respectively. The classification efficiency of the model was evaluated using a confusion matrix (Figure 1). The diagnostic performance of the XGBoost model was excellent, and except for six samples, it made accurate predictions in the other samples (Figure 2).

Details are in the caption following the image — **Figure 1**
Open in figure viewer PowerPoint

ML algorithm model and its confusion matrix. XGBoost algorithm model and its confusion matrix. A, Confusion matrix of the train set of the XGBoost algorithm model. B, Confusion matrix of the test set of the XGBoost algorithm model. The diagonal line from the top left to the bottom right is the number of correctly predicted samples.

In the XGBoost model, the total accuracy of the train and test sets was 88.63 and 84.21%, respectively. In the training set, the F1 values for positive and negative lymph node metastases were 0.85 and 0.91, respectively. In the test set, the F1 values for positive and negative lymph node metastases were 0.80 and 0.87, respectively. The AUC values of the ROC curve of the train and test sets were 0.92 and 0.82, respectively (Table 5).

Table 5. Evaluation of the XGBoost Model for Lymph Node Metastasis

	Cases	Accuracy	Recall Value	F1 Value	AUC	Cases	Accuracy	Recall Value	F1 Value	AUC
	Train Set					Test Set
Positive	33	0.85	0.85	0.85	0.92	14	0.75	0.86	0.80	0.82
Negative	55	0.91	0.91	0.91	0.92	24	0.91	0.83	0.87	0.82

We used the importance score to visualize the performance of the XGBoost prediction model (Supplemental Table 2). The importance score for CEA, tumor breadth, tumor length, ultrasound stage T, V_max, BMI, the location from the lower tumor margin to the upper internal anal sphincter, circumferential extent of the tumor, age, CA199, location from the anal margin, and RI was 0.2092, 0.1179, 0.0910, 2510.0910, 0.0758, 0.0709, 0.0701, 0.0607, 0.0556, 0.0531, and 0.0528, respectively, while the importance score for gender was 0. The bar graph (Figure 3) shows the predictive utility of each variable for lymph node metastasis in the algorithm application process. We found that CEA, tumor breadth, tumor length, and ultrasound staging had a greater impact on the diagnosis of lymph node metastasis in the XGBoost model. Figures 4 and 5 show two examples of correctly predicted lymph node metastasis and no metastasis, respectively. The waterfall plots (Figures 4F and 5F) demonstrate each variable's positive and negative effects on the predicted outcome in a single case. E[f(x)] represents the basic prediction probability of the XGBoost model, and f(x) represents the final prediction probability of the model.

The AUC value of the test set was 0.8214, while the corresponding value of radiologist A was 0.599, which indicated the diagnosis performance of the XGBoost model was better than that of radiologist A (Figure 6).

The accuracy, sensitivity, specificity, positive predictive value, and negative prediction value of the XGBoost model were 84.21, 75.00, 90.91, 85.71, and 83.33%, respectively. The accuracy, sensitivity, specificity, positive predictive value, and negative prediction value of 3D-ERUS alone were 65.87, 42.55, 79.75, 55.56, and 70.00%, respectively. This indicated that our XGBoost model had good performance compared with radiologist A (Table 6).

Table 6. Between-Group Comparisons of XGBoost Model and Radiologist A

	AUC	Accuracy	Sensitivity	Specificity	Positive Prediction	Negative Prediction
XGboost	0.8214	0.8421	0.7500	0.9091	0.8571	0.8333
Radiologist A	0.5990	0.6587	0.4255	0.7975	0.5556	0.7000

Discussion

Here we report on an XGBoost model for preoperative prediction of lymph node metastasis in patients with rectal cancer based on 3D-ERUS and related clinical information.

A study showed that the sensitivity and specificity of ERUS were 57 and 80%, respectively, while the values of CT were 79 and 76%.⁹ The 3D-ERUS can automatically capture and store high-resolution images of the rectal wall and cancer.^19-21 Thus, we used 3D-ERUS to evaluate lymph node metastasis. Compared with pathological examination, the sensitivity, specificity, and precision of ERUS were 42.55, 79.75, and 65.87%, respectively, which is consistent with previous reports.²²

An earlier study on predicting lymph node metastasis in rectal cancer mostly used both radiomics models and deep learning. Radiomics based on advanced pattern recognition tools has been found useful in extracting a large number of quantitative features from medical images. It can provide more metabolic and biological information than conventional imaging methods.²³ In a previous study, the least absolute shrinkage and selection operator (LASSO) regression model was constructed, which used feature selection based on the shear-wave elastography (SWE) ultrasomics signature of 87 patients.²⁴ The results of the study indicated the diagnostic accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the SWE ultrasomics signature were 82.8, 87.5, 78.8, 77.8, and 88.1%, respectively, while those of MRI were 75.9, 77.5, 74.5, 72.1, and 79.6%, respectively. A study created a radiomics nomogram based on T2-weighted imaging (T2WI) of 162 patients with rectal cancer, and this demonstrated good discrimination with an AUC of 0.937 for the training cohort and 0.884 for the validation cohort.²⁵ However, these were limited in that they used CT and MRI imaging, which mostly focused on the technical aspects of the algorithm and did not include clinical information such as age and tumor site. In our study, we found the XGBoost model based on the combined information, when compared with 3D-ERUS alone, showed better precision than radiologist A.

Machine learning has been widely used in healthcare. However, due to complex nonlinear relationships and unexplainable model results, some ML algorithms are difficult to interpret, resulting in a “black-box” problem that limits the clinical application of predictive models. In our study, we visualized the results of the ML algorithms and interpreted them by using Matplotlib and Seaborn. Matplotlib is an extended drawing library widely used in Python. It can generate high-quality graphics in a cross-platform interactive environment and quickly generate visual charts such as scatter charts, line charts, bar charts, grid graphs, and so on. Seaborn is an advanced visualization library based on Matplotlib.

Furthermore, different algorithms have different advantages. A study compared the predictive ability of logistic regression and some ML algorithms²⁶ and showed that the predictive ability of logistic regression was equally excellent. In our study, we found that the XGBoost algorithm was the best. We verified its prediction performance in comparison with other algorithms. XGBoost is a classification algorithm that allows the combination of several weak learning algorithms into a single strong learning algorithm with high performance.²⁷ The XGBoost prediction model has been extensively used to predict acute myocardial infarction^{28, 29} and preoperative microsatellite instability in rectal cancer.^{30, 31} In our study, the XGBoost model demonstrated a reliable preoperative prediction of lymph node metastasis in patients with rectal cancer.

In our study, there were significant differences between groups in tumor size (tumor length and tumor breadth), RI, periluminal diameter of invasion, and ultrasound T staging. Specifically, the risk of lymph node metastasis was positively correlated with tumor size, tumor RI, periluminal diameter of the invaded intestine, and ultrasound T stage. With regard to the predictive utility of specific parameters for lymph node metastasis, tumor RI and tumor length were the most important variables affecting lymph node metastasis. Consequently, the in situ status of the rectal tumor could influence the lymph node metastasis. Tumor RI may be related to neovascularization. Tumor RI was positively correlated with tumor invasiveness,³² with a large tumor size contributing to poor tumor morphology, increasing the likelihood of lymph node metastasis.

Serum CEA levels have shown prognostic value in colorectal cancer.³³ Specifically, high levels of CEA before surgery were associated with a poor prognosis for rectal cancer.³⁴ In our study, there was a significant between-group difference in serum CEA levels. In addition, CEA and CA199 showed high-importance scores in the model. Single indicators may be inadequate to inform individualized treatment; instead, it is important to comprehensively analyze patient information and identify influencing factors that can improve diagnostic accuracy.

This preliminary study established a prediction model based on 3D ultrasound examination, which showed significantly better prediction accuracy with simple data acquisition and modeling methods. Ultrasound is cheaper than MRI and CT and, additionally, involves less psychological discomfort and radiation exposure when compared with MRI and CT, respectively.

There are limitations to this study. First, this was a small-scale single-center retrospective study, and the findings need to be validated by further multicenter prospective studies. Second, some cases were eliminated due to the quality of the ultrasound images, which reduced the size of the sample. Future large-scale, multicenter cohort studies are warranted to validate our findings. Furthermore, future research should also consider an extensive analysis of image texture information, other imaging information (e.g., CT and MRI), and genetic factors when developing a model.

In conclusion, the XGBoost algorithm that we developed demonstrated significantly better preoperative prediction value in lymph node metastasis in patients with rectal cancer, and this can be helpful in determining clinical treatment strategies.

Open Research

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Supporting Information

Filename	Description
jum16297-sup-0001-FigureS1.jpgJPEG image, 240.8 KB	Supplemental Figure 1. Chart of patients eligible for the study.
jum16297-sup-0002-FigureS2.jpgJPEG image, 561 KB	Supplemental Figure 2. Receiver operating characteristic (ROC) curves of the machine learning algorithms (train set). A, Area under the ROC curve (AUC) of the train set of the RF algorithm model is 0.997. B, AUC of the train set of the LightGBM algorithm model is 0.7579. C, AUC of the train set of the SVM algorithm model is 0.9994. D, AUC of the train set of the XGBoost algorithm model is 0.9242.
jum16297-sup-0003-FigureS3.jpgJPEG image, 598.3 KB	Supplemental Figure 3. Receiver operating characteristic (ROC) curves of the machine learning algorithms (test set). A, Area under the ROC curve (AUC) of the test set of the RF algorithm model is 0.6682. B, AUC of the test set of the LightGBM algorithm model is 0.6518. C, AUC of the test set of the SVM algorithm model is 0.6057. D, AUC of the test set of the XGBoost algorithm model is 0.8214.
jum16297-sup-0004-Tables.docxWord 2007 document , 14.4 KB	Supplemental Table 1. Evaluation of the Machine Learning Algorithms Using 10-fold cross-validation. Supplemental Table 2. Importance value of each indicator.

Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.

References

1Bray F, Ferlay J, Soerjomataram I, Siegel RL, Torre LA, Jemal A. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 2018; 68: 394–424. https://doi.org/10.3322/caac.21492.
10.3322/caac.21492
PubMed Web of Science® Google Scholar
2Sung H, Ferlay J, Siegel RL, et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 2021; 71: 209–249. https://doi.org/10.3322/caac.21660.
10.3322/caac.21660
PubMed Web of Science® Google Scholar
3Anania G, Davies RJ, Arezzo A, et al. Rise and fall of total mesorectal excision with lateral pelvic lymphadenectomy for rectal cancer: an updated systematic review and meta-analysis of 11,366 patients. Int J Color Dis 2021; 36: 2321–2333. https://doi.org/10.1007/s00384-021-03946-2 [Erratum in: Int J Colorectal Dis (2021).
10.1007/s00384-021-03946-2
PubMed Web of Science® Google Scholar
4Ahmed M. Colon cancer: a clinician's perspective in 2019. Gastroenterol Res 2020; 13: 1–10. https://doi.org/10.14740/gr1239.
10.14740/gr1239
PubMed Web of Science® Google Scholar
5Weiser MR, Chou JF, Keshinro A, et al. Development and assessment of a clinical calculator for estimating the likelihood of recurrence and survival among patients with locally advanced rectal cancer treated with chemotherapy, radiotherapy, and surgery. JAMA Netw Open 2021; 4:e2133457. https://doi.org/10.1001/jamanetworkopen.2021.33457 [Errors in a Supplement. JAMA Netw Open 2022;5(1):e2147251. Published 2022 Jan 4.
10.1001/jamanetworkopen.2021.33457
PubMed Web of Science® Google Scholar
6Al-Sukhni E, Milot L, Fruitman M, et al. Diagnostic accuracy of MRI for assessment of T category, lymph node metastases, and circumferential resection margin involvement in patients with rectal cancer: a systematic review and meta-analysis. Ann Surg Oncol 2012; 19: 2212–2223. https://doi.org/10.1245/s10434-011-2210-5.
10.1245/s10434-011-2210-5
PubMed Web of Science® Google Scholar
7Park JS, Jang YJ, Choi GS, et al. Accuracy of preoperative MRI in predicting pathology stage in rectal cancers: node-for-node matched histopathology validation of MRI features. Dis Colon Rectum 2014; 57: 32–38. https://doi.org/10.1097/DCR.0000000000000004.
10.1097/DCR.0000000000000004
PubMed Web of Science® Google Scholar
8Zhuang Z, Zhang Y, Wei M, Yang X, Wang Z. Magnetic resonance imaging evaluation of the accuracy of various lymph node staging criteria in rectal cancer: a systematic review and meta-analysis. Front Oncol 2021; 11:709070. https://doi.org/10.3389/fonc.2021.709070.
10.3389/fonc.2021.709070
PubMed Web of Science® Google Scholar
9Pinto RA, Corrêa Neto IJ, Nahas SC, et al. Efficacy of 3-dimensional endorectal ultrasound for staging early extraperitoneal rectal neoplasms. Dis Colon Rectum 2017; 60: 488–496. https://doi.org/10.1097/DCR.0000000000000781.
10.1097/DCR.0000000000000781
PubMed Web of Science® Google Scholar
10Glynne-Jones R, Wyrwicz L, Tiret E, et al. Rectal cancer: ESMO clinical practice guidelines for diagnosis, treatment and follow-up. Ann Oncol 2018; 29:iv263. https://doi.org/10.1093/annonc/mdy161.
10.1093/annonc/mdy161
CAS PubMed Web of Science® Google Scholar
11Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med 2019; 380: 1347–1358. https://doi.org/10.1056/NEJMra1814259.
10.1056/NEJMra1814259
PubMed Web of Science® Google Scholar
12Li P, Song G, Wu R, et al. Multiparametric MRI-based machine learning models for preoperatively predicting rectal adenoma with canceration. MAGMA 2021; 34: 707–716. https://doi.org/10.1007/s10334-021-00915-2.
10.1007/s10334-021-00915-2
PubMed Google Scholar
13Bian J, Buchan I, Guo Y, Prosperi M. Statistical thinking, machine learning. J Clin Epidemiol 2019; 116: 136–137. https://doi.org/10.1016/j.jclinepi.2019.08.003.
10.1016/j.jclinepi.2019.08.003
PubMed Web of Science® Google Scholar
14Michael E, Ma H, Li H, Qi S. An optimized framework for breast cancer classification using machine learning. Biomed Res Int 2022; 2022:8482022. https://doi.org/10.1155/2022/8482022.
10.1155/2022/8482022
PubMed Web of Science® Google Scholar
15Liu X, Yang Q, Zhang C, et al. Multiregional-based magnetic resonance imaging radiomics combined with clinical data improves efficacy in predicting lymph node metastasis of rectal cancer. Front Oncol 2020; 10:585767. https://doi.org/10.3389/fonc.2020.585767.
10.3389/fonc.2020.585767
PubMed Web of Science® Google Scholar
16Bedrikovetski S, Dudi-Venkata NN, Kroon HM, et al. Artificial intelligence for pre-operative lymph node staging in colorectal cancer: a systematic review and meta-analysis. BMC Cancer 2021; 21: 1058. https://doi.org/10.1186/s12885-021-08773-w.
10.1186/s12885-021-08773-w
PubMed Web of Science® Google Scholar
17Beynon J, Foy DM, Roe AM, Temple LN, Mortensen NJ. Endoluminal ultrasound in the assessment of local invasion in rectal cancer. Br J Surg 1986; 73: 474–477. https://doi.org/10.1002/bjs.1800730618.
10.1002/bjs.1800730618
CAS PubMed Web of Science® Google Scholar
18Deng XH, Tang LN, Shen YH, Huang WQ, Chen YJ. Value of dual contrast-enhanced ultrasound in the preoperative T staging of rectal carcinoma. Zhonghua Yi Xue Za Zhi 2017; 97: 684–686. https://doi.org/10.3760/cma.j.issn.0376-2491.2017.09.010.
10.3760/cma.j.issn.0376-2491.2017.09.010
CAS PubMed Google Scholar
19Chen L, Liu X, Zhang W, et al. The predictive value of tumor volume reduction ratio on three-dimensional endorectal ultrasound for tumor response to chemoradiotherapy for locally advanced rectal cancer. Ann Transl Med 2022; 10: 666. https://doi.org/10.21037/atm-22-2418.
10.21037/atm-22-2418
PubMed Web of Science® Google Scholar
20Ren JH, Guo FJ, Dai WD, Han XJ, Ma N. Study of endorectal ultrasonography in the staging of rectal cancer. Chin Med J 2012; 125: 3740–3743.
PubMed Web of Science® Google Scholar
21Kim JC, Kim HC, Yu CS, et al. Efficacy of 3-dimensional endorectal ultrasonography compared with conventional ultrasonography and computed tomography in preoperative rectal cancer staging. Am J Surg 2006; 192: 89–97. https://doi.org/10.1016/j.amjsurg.2006.01.054.
10.1016/j.amjsurg.2006.01.054
PubMed Web of Science® Google Scholar
22Li XT, Sun YS, Tang L, Cao K, Zhang XY. Evaluating local lymph node metastasis with magnetic resonance imaging, endoluminal ultrasound and computed tomography in rectal cancer: a meta-analysis. Color Dis 2015; 17: O129–O135. https://doi.org/10.1111/codi.12909.
10.1111/codi.12909
PubMed Web of Science® Google Scholar
23Koçak B, Durmaz EŞ, Ateş E, Kılıçkesmez Ö. Radiomics with artificial intelligence: a practical guide for beginners. Diagn Interv Radiol 2019; 25: 485–495. https://doi.org/10.5152/dir.2019.19321.
10.5152/dir.2019.19321
PubMed Web of Science® Google Scholar
24Xian MF, Zheng X, Xu JB, Li X, Chen LD, Wang W. Prediction of lymph node metastasis in rectal cancer: comparison between shear-wave elastography based ultrasomics and MRI. Diagn Interv Radiol 2021; 27: 424–431. https://doi.org/10.5152/dir.2021.20031.
10.5152/dir.2021.20031
PubMed Web of Science® Google Scholar
25Li C, Yin J. Radiomics based on T2-weighted imaging and apparent diffusion coefficient images for preoperative evaluation of lymph node metastasis in rectal cancer patients. Front Oncol 2021; 11:671354. https://doi.org/10.3389/fonc.2021.671354.
10.3389/fonc.2021.671354
PubMed Web of Science® Google Scholar
26Christodoulou E, Ma J, Collins GS, Steyerberg EW, Verbakel JY, Van Calster B. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol 2019; 110: 12–22. https://doi.org/10.1016/j.jclinepi.2019.02.004.
10.1016/j.jclinepi.2019.02.004
PubMed Web of Science® Google Scholar
27Chen T, Guestrin C. XGBoost: a scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Acad. Med., (San Francisco, CA), 2016:785–794. https://doi.org/10.1145/2939672.2939785.
10.1145/2939672.2939785
Google Scholar
28AlAref SJ, Maliakal G, Singh G, et al. Machine learning of clinical variables and coronary artery calcium scoring for the prediction of obstructive coronary artery disease on coronary computed tomography angiography: analysis from the CONFIRM registry. Eur Heart J 2020; 41: 359–367. https://doi.org/10.1093/eurheartj/ehz565.
10.1093/eurheartj/ehz565
CAS PubMed Web of Science® Google Scholar
29Elfiky AA, Pany MJ, Parikh RB, Obermeyer Z. Development and application of a machine learning approach to assess short-term mortality risk among patients with cancer starting chemotherapy. JAMA Netw Open 2018; 1:e180926. https://doi.org/10.1001/jamanetworkopen.2018.0926.
10.1001/jamanetworkopen.2018.0926
PubMed Web of Science® Google Scholar
30Sheng S, Li R, Wang X, Gao HY, Zhang YH, Xu FQ. Predicting the 7th day efficacy of acupoint application ofChinese herbs(Xiao Zhong Zhi Tong tie) in patients with diarrhea – a machine-learning model based on XGBoost algorithm. World J Tradit Chin Med 2022; 8: 257–261. https://doi.org/10.4103/wjtcm.WJTCM_60_21.
10.4103/wjtcm.WJTCM_60_21
Google Scholar
31Zhang W, Huang Z, Zhao J, et al. Development and validation of magnetic resonance imaging-based radiomics models for preoperative prediction of microsatellite instability in rectal cancer. Ann Transl Med 2021; 9: 134. https://doi.org/10.21037/atm-20-7673.
10.21037/atm-20-7673
CAS PubMed Web of Science® Google Scholar
32Aung PP, Parra ER, Barua S, et al. B7-H3 expression in Merkel cell carcinoma-associated endothelial cells correlates with locally aggressive primary tumor features and increased vascular density. Clin Cancer Res 2019; 25: 3455–3467. https://doi.org/10.1158/1078-0432.CCR-18-2355.
10.1158/1078-0432.CCR-18-2355
CAS PubMed Web of Science® Google Scholar
33Thirunavukarasu P, Talati C, Munjal S, Attwood K, Edge SB, Francescutti V. Effect of incorporation of pretreatment serum carcinoembryonic antigen levels into AJCC staging for colon cancer on 5-year survival. JAMA Surg 2015; 150: 747–755. https://doi.org/10.1001/jamasurg.2015.0871.
10.1001/jamasurg.2015.0871
PubMed Web of Science® Google Scholar
34Huang CS, Chen CY, Huang LK, Wang WS, Yang SH. Prognostic value of postoperative serum carcinoembryonic antigen levels in colorectal cancer patients who smoke. PLoS One 2020; 15:e0233687. https://doi.org/10.1371/journal.pone.0233687.
10.1371/journal.pone.0233687
CAS PubMed Web of Science® Google Scholar

Volume42, Issue11

November 2023

Pages 2615-2627

Utility of Machine Learning Algorithms in Predicting Preoperative Lymph Node Metastasis in Patients With Rectal Cancer Based on Three-Dimensional Endorectal Ultrasound and Clinical and Laboratory Data