1 INTRODUCTION

Osteosarcoma is widely acknowledged as the most common type of primary malignant bone tumor that arises from mesenchymal tissue,¹ with an incidence rate of 4 to 5 cases per million people.² It has a bimodal age distribution pattern, occurring predominantly in children and young adults <20 years of age and those >50–60 years of age.^{3, 4} It has been reported that metastasis disease is found in approximately 20% of osteosarcoma patients at diagnosis.^{3, 5} Patients with localized osteosarcoma have a significantly higher 5-year survival rate than patients with distant metastasis (70% vs. 20%).⁶ Complete surgical resection combined with adjuvant chemotherapy is the best treatment for osteosarcoma patients with distant metastases.⁷ Despite this, the prognosis for metastatic disease remains dismal, with relapse happening in more than half of patients.8 Distant metastasis is widely considered as one of the most important long-term prognostic indicators for osteosarcoma patients. These findings highlight the need to develop technologies that can detect distant metastasis in patients at diagnosis to allow implementation of personalized therapeutic approaches.

Herein, we used the SEER Program database, a commonly used tool for analyzing rare cancers.⁹ The SEER Program is known to collect data from seventeen geographically diverse cancer registries, covering roughly 26% of the United States population. Modern data mining and machine learning approaches can assist in uncovering hidden correlations between parameters by extracting implicit relevant information and expertise from massive amounts of data.¹⁰ Over the years, machine learning has been successfully used to diagnose and predict many diseases.^{11, 12} Accordingly, we gathered a large, international cohort of osteosarcoma patients to develop and validate a machine learning-based model for analyzing clinical characteristics associated with the risk of distant metastasis.

2 METHODS

3 RESULTS

3.2 Feature analysis

Pearson correlation analysis was used to examine the relationship between each variable. The correlation heatmap (Figure 1) revealed a weak relationship between age and chemotherapy. Table 4 shows the results of machine learning classification algorithms applied on the balanced datasets generated with ADASYN techniques. Importantly, the ADASYN approach significantly enhanced the AP values of the classification models. The RF classifier achieved the highest precision values in the validation set for the 100% increase (0.688), the 150% increase (0.732) and the 200% increase (0.781) in validation set.

TABLE 4. Evaluation of the performance of classification models on imbalance dataset using ADASYN technique in validation set

Model	ADASYN	Precision	Accuracy (%)	Sensitivity (%)	Specificity (%)	F1-score
XGBoost	100%	0.655 (0.635–0.676)	0.693 (0.676–0.710)	0.699 (0.636–0.761)	0.706 (0.630–0.781)	0.619 (0.592–0.647)
	150%	0.694 (0.677–0.711)	0.687 (0.677–0.697)	0.696 (0.659–0.733)	0.707 (0.670–0.745)	0.659 (0.640–0.678)
	200%	0.757 (0.740–0.775)	0.711 (0.701–0.721)	0.675 (0.653–0.696)	0.763 (0.738–0.788)	0.681 (0.665–0.697)
LR	100%	0.553 (0.511–0.595)	0.614 (0.598–0.630)	0.666 (0.588–0.743)	0.618 (0.541–0.695)	0.550 (0.511–0.590)
	150%	0.547 (0.514–0.579)	0.616 (0.604–0.628)	0.572 (0.452–0.692)	0.661 (0.584–0.737)	0.547 (0.481–0.613)
	200%	0.663 (0.653–0.673)	0.613 (0.599–0.628)	0.687 (0.613–0.761)	0.571 (0.514–0.628)	0.626 (0.588–0.663)
RF	100%	0.688 (0.666–0.711)	0.703 (0.692–0.714)	0.623 (0.572–0.673)	0.800 (0.755–0.845)	0.595 (0.569–0.621)
	150%	0.732 (0.717–0.746)	0.708 (0.697–0.720)	0.690 (0.678–0.701)	0.746 (0.735–0.758)	0.671 (0.660–0.681)
	200%	0.781 (0.766–0.795)	0.718 (0.710–0.726)	0.687 (0.659–0.715)	0.781 (0.751–0.812)	0.688 (0.667–0.708)
GaussianNB	100%	0.575 (0.556–0.594)	0.609 (0.592–0.627)	0.453 (0.374–0.531)	0.819 (0.743–0.895)	0.458 (0.411–0.504)
	150%	0.626 (0.604–0.647)	0.631 (0.616–0.647)	0.635 (0.563–0.707)	0.641 (0.565–0.718)	0.614 (0.565–0.662)
	200%	0.661 (0.640–0.682)	0.619 (0.609–0.628)	0.553 (0.450–0.656)	0.710 (0.609–0.812)	0.576 (0.505–0.646)
kNN	100%	0.636 (0.620–0.651)	0.692 (0.681–0.703)	0.705 (0.629–0.781)	0.680 (0.606–0.753)	0.623 (0.588–0.658)
	150%	0.684 (0.670–0.698)	0.698 (0.687–0.709)	0.659 (0.591–0.727)	0.737 (0.672–0.802)	0.665 (0.629–0.701)
	200%	0.717 (0.707–0.727)	0.697 (0.693–0.702)	0.740 (0.693–0.787)	0.668 (0.623–0.714)	0.719 (0.695–0.743)
SVM	100%	0.631 (0.611–0.652)	0.646 (0.630–0.662)	0.664 (0.606–0.722)	0.655 (0.595–0.716)	0.575 (0.544–0.606)
	150%	0.644 (0.629–0.659)	0.643 (0.629–0.658)	0.652 (0.574–0.730)	0.658 (0.577–0.739)	0.608 (0.569–0.647)
	200%	0.678 (0.658–0.699)	0.637 (0.631–0.643)	0.661 (0.596–0.725)	0.624 (0.560–0.689)	0.628 (0.599–0.657)

3.3 Model development and evaluation

Six machine learning algorithms were employed to build a prediction model in this study. The training set was used to create and train machine learning models. After adjusting for parameters and comparing algorithms, the average precision values of six machine learning models were greater than 0.64, indicating that predictive models have good predictive ability. (Figure 2). As described in Figure 2, the RF algorithm yielded the best prediction performance in precision-recall curves between the training and validation set. The precision-recall curves, calibration plots, and DCAs for the validation set were generated to evaluate the prediction model. The calibration plots of the validation set indicated that the predictive probabilities against observed the risk of distant metastasis (Figure 3A). DCA of the six models was subsequently conducted, showing that all models achieved net clinical benefit against a treat all-or-none plan, except kNN (Figure 3B). Regarding calibration plots and DCAs, the RF model also performed best. Table 4 shows the confusion matrix evaluation measures as well as the average precision of all prediction models. All prediction models' k-fold cross-validation accuracies (k = 10) are listed in Table 5. Our results showed that the RF model yielded the highest k-fold cross-validation accuracy. Among these, the predictive model using the RF algorithm yielded the best predictive performance. (Figure 4).

TABLE 5. The k-fold cross-validation accuracies (k = 10) of all six prediction models

Model	XGB	RF	Gaussian NB	kNN	SVM	LR
k-fold accuracy	0.711 (0.701–0.721)	0.718 (0.710–0.726)	0.619 (0.609–0.628)	0.697 (0.693–0.702)	0.637 (0.631–0.643)	0.613 (0.599–0.628)

3.4 Model interpretation

Figure 5A depicts the predictive model's SHAP summary plot, which consists of 11 features sorted by their impact on metastatic status. The higher the SHAP value of a feature, the higher the risk of distant metastasis. The red color denotes a high feature value, purple denotes a feature value close to the overall average, and blue denotes a low feature value. Figure 5B provides an example for predicting the risk of metastasis in an osteosarcoma patient. In this case, the RF model predicted a distant metastasis risk of 1.00 (base value: 0.35). The probability of distant metastasis was increased by tumor size (more than 10 cm), radiation therapy, and lack of surgical intervention. The axial placement of the tumor decreased the estimated probability of metastasis. The axial tumor location reduced the predicted risk of metastasis.

4 DISCUSSION

Patients with osteosarcoma who develop distant metastasis have a significantly lower overall survival rate, with a 5-year survival rate of only 20%.¹⁶ It has been established that patients with distant metastasis derive little benefit from surgery, chemotherapy, or novel immunotherapy, accounting for the poor prognosis of advanced osteosarcoma patients.¹⁷ As a result, it is critical to identify risk factors associated with distant metastasis in osteosarcoma patients to facilitate early detection, prevention, and prognosis assessment.

An ML-based model was trained and validated in this study to predict the risk of distant metastasis. This ML model was based on clinical data, including age, gender, race, tumor size, primary tumor site, grade, histology, surgery, radiotherapy, chemotherapy, lymphadenectomy, and stage. The ADASYN technique was then used to deal with imbalanced data, yielding a more balanced dataset. As a result, the RF model outperformed all other models in training and validation sets. To improve the interpretability of our machine learning model, we reported the SHAP values and a list of highly influential features.

We see at least two approaches to clinical practice implementation. To begin, individual doctors and patients could use the model to assess risk and make tailored decisions about osteosarcoma screening in the clinic. Second, health systems could use the model to identify high-risk patients for additional outreach at the population level.

In the present study, SHAP analysis demonstrated that the top three informative features in the model were tumor size, primary surgery, and age. The present study consistently showed that tumor size is an important risk factor. Moreover, it has been reported that a large tumor at the time of diagnosis is associated with a greater number of lung metastases.¹⁸ Interestingly, tumor size has been shown to be an independent predictor of overall survival and distant metastases in patients with osteosarcoma using plain radiographs or magnetic resonance imaging.⁹ In this regard, tumors treated without surgery, with higher pathological grades and larger size, contributed to an increased risk of cancer metastasis. Surgery, which involves removing the tumor or amputating, remains the only effective treatment for osteosarcoma and is not indicated in patients with distant metastasis.¹⁹ Although the efficacy of surgery is variable due to various factors, radical resection of the primary tumor inhibits tumor progression, including metastases, to a certain extent.^{20, 21} Our findings indicate that the absence of surgery exerted a significant negative effect on distant metastasis, which corroborates the above findings. Although age as risk factor for distant metastases are less intuitive than primary surgery and tumor size has been previously found to be risk factors for advanced disease in in studies of other malignancies.^{22, 23} It had been found in a large study that includes over 27,000 patients that older age was an independent significant risk factor for distant metastases.²⁴ Similarly, a recent study had also discovered that older age at diagnosis was at a higher risk of developing pleural metastases.²⁵ In concordance with these studies, our results find that a high risk of distant metastasis among patients older than 60 years. In this study, we divided patients into separated into different age groups (<24 years old, 24–59 years old, and >60 years old) according to recent publication.²⁶ The reasons for this may be related to health access, among other socioeconomic features.

Previous efforts to improve osteosarcoma prognostic prediction were based on parametric regression models, widely used in clinical studies due to their ease of use and interpretability. Regression analysis, on the other hand, can only be carried out in its entirety.²⁷ Importantly, regression modeling assumes linear and homogeneous relationships among input variables. Nonetheless, it should be borne in mind that complex interactions exist between predictors.²⁸ Decision tree-based methods are a subset of machine learning algorithms that uncover complex non-linear relationships between covariates.²⁹ The final class of an instance in a random forest is determined by outputting the class that is the mode of individual tree outputs, resulting in robust and accurate classification as well as the ability to handle a large number of input variables.³⁰ Random forests are relatively resistant to overfitting and are capable of handling datasets with highly asymmetric class distributions.³¹ In the training and validation sets, our AI model based on RF had a high prediction performance for the risk of distant metastasis in osteosarcoma patients (AP = 0.903 and 0.781, respectively). The retrospective nature of this study, however, limited it, and selection bias was inevitable. Moreover, we could not identify socioeconomic factors linked to patient survival and the occurrence of pathological fractures in osteosarcoma. Finally, “no” and “unknown” were combined into one group in the SEER data for chemotherapy and radiation. Chemotherapy and radiation were significantly underreported, which we could not overlook.

In conclusion, we employed the random forest method to develop an artificial intelligence model for predicting the probability of distant metastasis in patients with osteosarcoma. According to precision-recall analysis, our prediction model yielded accurate results. DCA demonstrated that the model provided a net benefit. Our findings substantiate that this prediction model has huge prospects for application during clinical practice to assist physicians in making more informed treatment decisions for osteosarcoma patients.

AUTHOR CONTRIBUTIONS

Conceptualization, BL.B. and Q.Y.; Methodology, ZY.W. and SJ.W.; Software, SJ.W.; Investigation, BL.B.;Resources, BL.B. and Q.Y.; Data curation, Q.Y.; Formal analysis, ZY.W., SJ.W. and BL.B.; Validation, BL.B. and Q.Y.; Writing-original draft preparation, ZY.W., SJ.W., and Q.Y.; Writing—review and editing, BL.B. and Q.Y.; Visualization, BL.B.; Supervision, Q.Y.

FUNDING INFORMATION

This research did not receive any specific grant from funding agencies in the public, commercial, or not- for- profit sectors.

CONFLICT OF INTEREST

We have read and understood Cancer Medicine's policy on disclosing conflicts of interest and declare that we have none.

ETHICAL APPROVAL STATEMENT

We received permission to access the research data file in the SEER program from the National Cancer Institute, US. Approval was waived by the local ethics committee, as SEER data are publicly available and de-identified.