1 BACKGROUND

Kidney cancer is a malignant tumor that originates from the urinary tubule epithelial system of the renal parenchyma, accounting for approximately 20% of all urinary system tumors [1]. As of 2018, the number of new cases of kidney cancer worldwide exceeded 400,000, and the number of deaths exceeded 170,000 [2]. In China, the annual incidence of kidney cancer is increasing. In 2016, the incidence of kidney cancer exceeded 15,000, and deaths exceeded 5000. The incidence rate was 4.02 per 100,000, and the mortality rate was 1.37 per 100,000 [3]. The surveillance, epidemiology, and end results (SEERs) Program provides information on cancer statistics to reduce the cancer burden among the US population. This program was supported by the surveillance research program (SRP) of the NCI's Division of Cancer Control and Population Sciences (DCCPS) [4]. The data from the SEER database is collected from the cancer registry of the population and currently covers 30% of the population data in the United States. Globally, about half of kidney cancer cases are diagnosed at age 65, with peak incidence at age 75 [5]. Early-stage localized kidney cancer have a good prognosis through radical nephrectomy, but patients with stage II or III have a high risk of recurrence after nephrectomy [5]. Immunotherapy has become the most promising treatment for patients with metastatic kidney cancer, as kidney cancer is resistant to chemotherapy [6, 7]. But immunotherapy is only successful in 10%–15% of patients [7]. Therefore, treatment for metastatic kidney cancer remains inadequate.

Renal cell carcinoma is the most common malignant renal tumor. Risk prediction models such as Memorial Sloan-Kettering Cancer Center (MSKCC), International Metastatic Renal-Cell Carcinoma Database Consortium (IMDC), Leibovich, University of California, and Los Angeles Integrated Staging System (UISS) have been applied to identify patients at high risk of recurrence after surgery [8]. Traditional statistical analysis is a basic description of sample data based on assumption of sample distribution, and then make inferences in the form of probability, and it is mainly inferred through the observation of a random process in a limited period of time. Although the predictive capabilities have been improved, it is still far from meeting the clinical needs for patient classification accurately. In contrast, based on rich data, machine learning methods predict future unknown outcomes by generalizing features without making assumptions about data distribution or understanding the mechanisms behind sample data. Due to the complexity and diversity of a large amount of cancer data, traditional statistical inference models are facing challenges when processing and analyzing it effectively or efficiently; however, machine learning methods demonstrated advantages for analyzing huge volume and high complexity data. For example, Byun et al. used deep learning method to predict the prognosis in nonmetastatic clear cell renal cell carcinoma [9]. Machine learning models also showed good performance in some survival prediction analyses for other cancer types. Gu-Wei Ji et al. used machine learning models to predict the recurrence of hepatocellular carcinoma after resection [10].

Therefore, we incorporate machine learning models into this study to explore a more efficient, intelligent, and accurate prediction model for the prognosis and metastasis risk assessment of kidney cancer patients.

2 MATERIALS AND METHODS

2.1 Patient data set

We included patients with histologically diagnosed kidney cancer who had complete survival time and active follow-up data from 2004 to 2015 in the SEER database. The data selection process was illustrated in Figure 1. Kidney and renal pelvic cancers were described together in the SEER database. We chose the disease with ICD-10-CM code C64.9: Malignant neoplasm of the unspecified kidney, except the renal pelvis. Patients diagnosed with cancer only through autopsy or death certificates, patients had missing follow-up records and patients with pathological features of “Blank” “N/A” “Unknown” were excluded. Tumor staging was coded according to the sixth edition of the TNM staging system of the American Joint Committee on Cancer of the United States.

3 RESULTS

3.3 Performance of prognostic models

Eight machine learning models including SVM, logistic regression, decision tree, random forest, XGBoost, AdaBoost, KNN, and multilayer perceptron were developed to predict the survival and metastasis of kidney cancer patients. Accuracy, precision, recall (sensitivity), specificity, F1 score, and area under the ROC curve (AUC) were applied used to evaluate the model performance. The results showed that logistic regression had the best performance in terms of AUC of 0.741 for 3-year survival and AUC of 0.804 for metastasis, respectively (Figures 2 and 3). For the 3-year survival predictive study, logistic regression model had the accuracy of 0.684, the sensitivity of 0.702, the specificity of 0.670, the precision of 0.686, and the F1 score of 0.683. For the tumor metastasis predictive study, the Logistic regression model had the accuracy of 0.800, the sensitivity of 0.540, the specificity of 0.830, the precision of 0.769, and the F1 score of 0.772 (Tables 3 and 4). When we stratified the population by histological type, the patients with clear cell renal carcinoma demonstrated the similar findings (Table 5). For the 3-year survival, logistic regression model had the AUC of 0.710, the accuracy of 0.658, the sensitivity of 0.674, the specificity of 0.649, the precision of 0.661, and the F1 score of 0.653. For the metastasis, logistic regression model had the AUC of 0.811, the accuracy of 0.826, the sensitivity of 0.593, the specificity of 0.851, the precision of 0.802, and the F1 score of 0.803.

Table 3. Model performance on 3-year survival prediction

Model	Accuracy	Sensitivity	Specificity	Precision	F1-score	AUROC
Logistic regression	0.684	0.702	0.670	0.686	0.683	0.741
Random forest	0.645	0.654	0.638	0.643	0.642	0.690
XGBoost	0.676	0.665	0.692	0.678	0.674	0.729
Decision tree	0.690	0.733	0.662	0.697	0.687	0.710
K-nearest neighbors	0.607	0.609	0.605	0.607	0.606	0.609
AdaBoost	0.675	0.675	0.675	0.675	0.675	0.736
Support vector machines	0.685	0.713	0.665	0.689	0.683	0.684
Multilayer perceptron	0.684	0.732	0.654	0.693	0.679	0.735

• Abbreviation: AUROC, Area under the receiver operating characteristic.

Table 4. Model performance on tumor metastasis prediction

Model	Accuracy	Sensitivity	Specificity	Precision	F1-score	AUROC
Logistic regression	0.800	0.540	0.830	0.769	0.772	0.804
Random forest	0.765	0.417	0.831	0.744	0.753	0.731
XGBoost	0.806	0.548	0.842	0.781	0.786	0.747
Decision tree	0.791	0.498	0.828	0.759	0.765	0.666
K-nearest neighbor	0.763	0.403	0.826	0.738	0.747	0.682
AdaBoost	0.797	0.518	0.846	0.777	0.784	0.799
Support vector machines	0.793	0.510	0.818	0.754	0.755	0.573
Multilayer perceptron	0.800	0.542	0.828	0.768	0.770	0.802

• Abbreviation: AUROC, Area under the receiver operating characteristic.

Table 5. The Logistic Regression performance on 3-year survival and tumor metastasis prediction for clear cell renal cell carcinoma

Performance on 3-year survival prediction		Performance on tumor metastasis prediction
AUROC	0.710	AUROC	0.811
Accuracy	0.658	Accuracy	0.826
Sensitivity	0.674	Sensitivity	0.593
Specificity	0.649	Specificity	0.851
Precision	0.661	Precision	0.802
F1-score	0.653	F1-score	0.803

• Abbreviation: AUROC, Area under the receiver operating characteristic.

4 DISCUSSION

The incidence of kidney cancer has been increasing annually. Renal cell carcinoma is the most common type of kidney cancer. Most localized renal cell carcinomas had a good prognosis after cytoreductive nephrectomy, but approximately 20%–30% of patients died due to the recurrence and metastasis after surgery. Precisely prediction the clinical outcome for early diagnosis and treatment are essential to improve the prognosis of kidney cancer.

Clear cell renal cell carcinoma (ccRCC) accounts for 60%–80% of all renal cell carcinomas [23]. ccRCC has a better clinical prognosis than other types of kidney cancer. In our multivariate logistic regression model for survival and metastasis, the ccRCC patients showed better outcomes with an odds ratio 0.897 (95% confidence interval: 0.825–0.975, p = 0.01) for survival and an odds ratio 1.372 (95% confidence interval: 1.235–1.525, p < 0.001) for metastasis compared to other histological types. ccRCC is characterized by loss of chromosome 3p and biallelic inactivation of the VHL gene due to mechanisms such as loss of heterozygosity (LOH), mutation, and methylation [24]. High-frequency mutations of genes (PBRM, SETD2, BAP1, etc.) involved in epigenetic regulation were also found in more than 50% of ccRCC [24]. In this –omics era, biomarkers especially the genetic biomarkers has been widely applied in clinical area. The genetic signature has been incorporated in risk prediction models, such as ClearCode34 and 16-gene assay [25, 26]. In the future, the genetic biomarkers should be incorporated in our current model and be evaluated in an independent population.

Clinical prediction models are commonly used to help clinicians and patients make better medical decisions, and to help government departments and health managers better manage medical quality and rationally allocate medical resources. In recent years, with the advance of science and technology, cancer diagnosis and prediction has entered a multidimension era of big data. The conventional clinical data measurements include images, genes, blood, proteins, pathological analysis, etc., which bring great challenges to the data analysis and disease diagnosis, and due to various reasons, some data will be partially missing. At present, the methods of cancer prediction are mainly divided into traditional statistical inference and machine learning models. The former is the basic description of the sample data based on the assumption of the sample distribution, and then inferences in the form of probabilities. And it is mainly inferred by observing a random process in a limited period of time. Machine learning models are very different from statistical inferences in that they take advantage of rich data and learn new patterns through generalized features to predict future outcomes without the need to make assumptions about the distribution of the data and understand the mechanisms behind the sample data. Due to the complexity and diversity of large amounts of data, traditional statistical inference models are less accurate than machine learning models, and the general applicability is not as strong as the latter, so modern cancer predictions are gradually incorporated into machine learning models. Machine learning also has its inherent problems, such as the inability to judge the advantages and disadvantages of the model from the algorithm, and the results are difficult to interpret, so these problems need to be evaluated and overcome through external verification before actual use.

Eight machine learning models were applied in this study. SVMs, also support-vector networks are supervised learning models with associated learning algorithms that analyze data for classification and regression analysis [11]. Logistic regression is a statistical model that in its basic form uses a logistic function to model a binary dependent variable, although many more complex extensions exist. In regression analysis, logistic regression is estimating the parameters of a logistic model (a form of binary regression) [12]. Decision trees and closely related influence diagrams are used as visualization and analysis decision support tools, in which the expected value (or expected utility) of competing solutions is calculated [13]. Random forests or random decision forests are an ensemble learning method for classification, regression, and other tasks that operates by constructing a multitude of decision trees at training time [14]. XGBoost aims to provide a “scalable, portable, and distributed gradient boosting (GBM, GBRT, and GBDT) library.” It runs on a single machine, along with distributed processing frameworks Apache Hadoop, Apache Spark, Apache Flink, and Dash [15]. AdaBoost, is a statistical classification meta-algorithm that can be used in conjunction with many other types of learning algorithms to improve performance, and often referred to as the best out-of-the-box classifier [16]. K-nearest neighbors (KNN) is a nonparametric classification method used for classification and regression. In both cases, the input consists of the k training examples closest to the data set. The output depends on whether k-NNs are used for classification or regression [17]. Multilayer perceptions (MLP) is a class of feed-forward artificial neural networks (ANNs) that consist of at least three layers of nodes: the input layer, the hidden layer, and the output layer. Except for the input nodes, each node is a neuron that uses a nonlinear activation function. MLP is trained using a supervised learning technique called back propagation [18]. Its multilayer and nonlinear activation distinguish MLP from linear perceptions. It can distinguish nonlinearly separable data [18]. We compared the performance of these eight models, there were no significant difference in performance. It is hard to compare the models based on its algorithm, external validation should be conducted for model evaluation.

In addition, we used a single source of public data to ensure data completeness, which had limited generalizability. Although the variable selection was based on statistics and clinical experience, the overfitting was a common problem in machine learning models. We will subsequently optimize and balance the number of variables and the prediction results. In the future, this prediction models could be externally validated using hospital patient data. Furthermore, in real clinical scenarios, these clinical prediction models need to be transformed into useful tools after strong clinical validation, and they will be integrated into the hospital information system to provide timely decision support for future cancer care.

5 CONCLUSION

The present study developed and compared eight machine learning models for predicting 3-year survival and metastasis of kidney cancer. The variable selection was based on bother statistical significance and clinical experiences. In conclusion, Logistic Regression had the highest area under the receiver operating characteristic (AUROC) in both 3-year survival and metastasis prediction scenarios, and the prediction results potentially provide decision-making references to clinicians for early intervention and treatment.

AUTHOR CONTRIBUTIONS

Yuxiang Zhang: Methodology; writing—original draft; writing—review and editing. Na Hong: Data curation; methodology; writing—review and editing. Sida Huang: Formal analysis. Jie Wu: Methodology. Jianwei Gao: Data curation; software. Zheng Xu: Software. Fubo Zhang: Software. Shaohui Ma: Writing—original draft. Ye Liu: Writing—original draft. Peiyuan Sun: Software. Yanping Tang: Supervision. Chun Liu: Data curation; software. Jianzhong Shou: Writing—review and editing. Meng Chen: Conceptualization; project administration; resources; supervision; writing—review and editing.

CONFLICT OF INTEREST

All authors declare that there is no conflict of interest except Professor Meng Chen, who is member of Cancer Innovation Editorial Board. To minimize bias, she was excluded from all editorial decision-making related to the acceptance of this study for publication.

ETHICS STATEMENT

Not applicable.

INFORMED CONSENT

Not applicable.