1 INTRODUCTION

Venous thromboembolism (VTE) comprises both deep-vein thrombosis (DVT) and pulmonary embolism (PE) [1]. The association between VTE and cancer is well-established with cancer patients accounting for approximately 20% of all VTE incidents [2]. While the estimated prevalence of VTE in the general population is around 1 in 1000 [3, 4], some estimates suggest this number increases fivefold within the cancer patient population [1, 5, 6]. The risk increases further among patients who receive chemotherapy as shown in a 15-year population-based study [7].

VTE is a multifaceted risk in cancer patients that exacerbates clinical consequences, significantly impacting morbidity, mortality, and cost of patient care [1, 5, 8-11]. Specifically, VTE-associated mortality is 2.2 times more likely in VTE patients with cancer than in those without [10]. VTE is the leading cause of mortality in cancer patients, aside from mortality due to cancer itself [1, 8]. In addition to increasing risk of mortality, VTE burdens the cancer treatment process. When managing VTE in cancer patients, use of anticoagulants, which thin the blood, requires rigorous patient monitoring to achieve adequate anticoagulation and to identify complications such as bleeding. Compared with cancer patients without VTE, patients with VTE have over two times the risk of experiencing major bleeding [12]. Bleeding can worsen anemia while reduced blood counts can delay cancer interventions such as chemotherapy and radiotherapy and increase the need for blood transfusions.

The recurrence rates of VTE are also high in patients with cancer. Patients with an active malignancy have a three- to fourfold higher risk of recurrence compared with patients without cancer, and the risk is further increased in those with metastatic cancers. According to one study, the 1-year cumulative risk for recurrent VTEs after the first episode was 21% in cancer patients compared with 7% in patients without cancer [12]. All the VTE-related factors discussed above can affect cancer management, increase treatment costs, and escalate average price per hospitalization for cancer patients [2, 3, 12, 13].

Treatments such as anticoagulant therapy are available, both for prophylaxis against occurrence, as well as for treatment of VTE in cancer patients. Appropriate and timely use of the prophylactic measures are vital for reducing the risk of both fatal and nonfatal PE as well as the postthrombotic complications [14]. Anticoagulants are drugs that interfere with blood coagulation cascade to reduce or inhibit blood clotting. The low-molecular-weight heparin (LMWH) has been found in multiple studies to reduce the likelihood of a VTE event occurring in a cancer patient [2, 15-17]. With these issues in mind, it is evident that effective VTE prophylaxis in cancer patients has the potential to drastically improve cancer survival rates and decrease treatment costs for hospitals and patients alike. However, while anticoagulant prophylaxis and treatment is effective in primary and secondary prevention of VTE, as mentioned above, there are certain implications with their regular use in all cancer patients. In particular, anticoagulants are associated with increased bleeding, require parenteral administration, training, and additional monitoring, all of which can increase both cost and complexity of cancer patient management [2, 12, 18]. Therefore, it is important to stratify and define high-risk cohorts of cancer patients who are prone for VTE. There is thus a need for effective VTE risk stratification systems to ensure that prophylaxis is administered only to high-risk patients. An accurate, reliable, and robust VTE stratification system would help clinicians in decision-making about anticoagulant therapy at the point of care (POC). Prophylactic measures against VTE are often implemented for hospitalized patients, so high-risk stratification is particularly important in ambulatory patients (outpatients) as they cannot be monitored as closely as hospitalized patients.

The importance of delineating which cancer patients are at increased risk of VTE for instituting anticoagulation prophylaxis, particularly ambulatory patients, is critical as anticoagulation is associated with significant risks and costs in already debilitated cancer patients. Decision to provide prophylactic anticoagulation in ambulatory patients clinically alone is often difficult and providers need a decision support tool that pinpoints the most vulnerable groups for VTE. Several cancer-associated thromboembolism (CAT) prediction scores have been developed, such as Khorana [19], Vienna CATS [20], PROTECHT [21], and CONKO [22] based on routinely collected patient care data. These risk-assessment methods all use a simple scoring system where points are added based on each of five to eight different predictors with higher scores indicating a higher risk of developing VTE. Some of the predictors that these scores use include cancer site, platelet count, white blood cell count, hemoglobin, use of red blood cell stimulating factors, and body mass index (BMI). Of these scores, the Khorana score is the most validated and used [23]. However, despite its acceptance in the research community, the Khorana score still only achieves a positive predictive value of 6.7%, which is not meaningful enough to make a quantified decision by the clinicians and thus leaves plenty of room for improvement [19]. In another study of 218 patients with cancer-initiating chemotherapy, it is shown that the Khorana score was able to stratify ambulatory cancer patients according to the risk of VTE, but not for all cancer types [24]. The Khorana score can be used to select ambulatory cancer patients at high risk of VTE for thromboprophylaxis, but most events occur outside this high-risk group [25].

During informal discussions, clinicians opined that even a positive predictive value of 20%–30% will help them with decision-making, tipping the decision one way or other with some scientific qualitative basis, and those discussions motivated the team to explore various features (clinical biomarkers) and develop more robust and clinically meaningful predictive models.

In this study we use machine learning to take a data-driven approach to VTE prediction in cancer patients. Our aim in this study is to not only improve upon the performance of known risk assessment scores such as the Khorana score but also to perform an in-depth, data-driven exploration of both new and known VTE risk factors.

Traditional approaches to prediction in medicine often focus on capturing medical expertise through a set of carefully designated rules [26]. However, data-driven approaches, such as machine learning algorithms instead can learn effective prediction decisions by observing numerical patterns in the input data [26, 27]. One subset of machine learning, known as supervised learning, deals with training a model to accomplish this task of classifying data based on a set of input data with labeled ground truth values [27]. Supervised learning has the advantage over traditional rule-based methods of being able to leverage computational power to identify highly convoluted patterns in massive datasets with large numbers of potential predictors relatively quickly and efficiently [26, 28]. Such an approach has promise in the context of cancer patient VTE prediction, where the currently accepted scoring systems are simple rule-based methods that do not necessarily capture a wide range of the potentially complex interactions between variables [19, 20, 22]. Ferroni et al. have designed a precision medicine approach to exploit significant patterns in data to produce VTE risk predictors for cancer outpatients [29]. They have used multiple kernel learning (MKL) [30] based on support vector machines (SVM) models to predict VTE risk. In our research, we have examined VTE classification performances of several standard machine learning (ML) algorithms including SVM, logistic regression (LR), and Random Forest (RF) and compared these to the baseline performance of the Khorana score.

Methods and results are described in the following sections.

2 METHODS

In this retrospective study of a population of cancer patients at the University of California Davis Medical Center (UCDMC), we used ML to explore both new and known VTE risk factors. Our goal was to not only develop a machine-learning-based VTE risk assessment system for cancer patients but also to examine which risk factors may be useful when taking such an approach. From our efforts, we hope to establish a foundation for using machine learning to eventually answer more complex questions about VTE prediction in cancer patients, such as how changes in a patient's condition, as the patient continues with his/her cancer management, affect the risk of developing VTE over time.

In this study, we examined 29 features in total, including a selection of available features from the Khorana score and biomolecular markers from a previous study of CAT [19, 31]. Since relevant VTE events can occur before or after cancer diagnosis and clinical interventions (i.e., surgery, chemotherapy, radiotherapy), we used a set of time-agnostic features to gain a view of how a patient's general profile over a large period of time may or may not be indicative of VTE risk. Each of the features we used covered information about a patient's background, cancer, lab values, or medications.

We then explored the utility of our feature set in a machine learning context in a two-phased approach. In the first phase, we trained several different models with a spectrum of hyperparameter choices on four different feature subsets that were derived both from performed feature selection experiments and from pre-determined feature pools. We then identified the best-performing model and feature set combination and, in a second phase of experiments, attempted to reduce the number of used features without sacrificing performance through an iterative feature accumulation process. Finally, we validated the performance of our chosen model on a held-out data set extracted from our original data.

3 RESULTS

3.3 Model optimization

For the second phase of our study, we looked at optimizing the feature set for our best-performing model configuration and validating the performance on our held-out test set. Based on the results presented in Table 4, we used the 500-tree RF model trained on our entire feature pool as a baseline for our best-performing model. In this section, we present the results of using this model with the previously described wrapper feature selection method to reduce the dimensionality of the feature set while attempting to maintain the same level of model performance.

Table 4. Cross-validation of 500-tree random forest (mean ± SD) on all feature sets.

	All (n = 29)	Khorana (n = 5)	Clinical (n = 5)	Filtered (n = 20)	Wrapper (n = 15)
Random forest (500 trees)	0.761 ± 0.065	0.684 ± 0.073*	0.696 ± 0.071*	0.755 ± 0.067	0.769 ± 0.072

• * p < 0.05 from DeLong test when compared with model trained on all features (first column).

3.3.1 Wrapper selected features

Table 4 compares the cross-validation performance of the 500-tree RF model using the wrapper-selected feature set to the results from the first phase of the study. When compared with the model trained on all features, the wrapper and filtered feature sets are the only feature sets that did not result in a statistically significant decline in performance. This confirms that the wrapper method was effective in identifying a reduced subset of features (52% of the whole feature pool and 75% of the filtered feature pool), without sacrificing performance.

The ordered list of features accumulated when performing the wrapper feature selection method with the RF model of 500 trees are:

creatinine serum, antineoplastic-aromatase inhibitors, MCHC, red blood cell count, stage, immunosuppressives, antineoplastic-antiandrogenic agents, protein, site, MCV, antineoplastic-alkylating agents, albumin, antineoplastic-antimetabolites, MCH, histopathological type.

The curve illustrated in Figure 1 shows the relationship between these features and the AUROC of our model during feature accumulation. Each model evaluation came from the average result of 10 iterations of 10-fold cross-validation. The x-axis represents each iteration of the recursive accumulation of features, while the y-axis represents the AUROC associated with the model trained after each added feature. The model trained on this set of recursively selected features not only matched the performance of the model trained on all features with no statistical difference between ROC outputs, but also did so with only 15 features, reducing the size of our feature set by 14. The ROC and PRC curves resulting from training a model on these 15 features are contained in Figures 3 and 4, respectively.

Unlike in the clinical expert and filter-selected feature sets, seven different medications were included in the wrapper-selected feature set, although only two appeared in the first twelve selected features. Furthermore, the white blood cell count and platelet count lab tests were excluded despite being included in both of our other examined feature sets as well as the Khorana score. This exclusion is not to undermine the usefulness of the features to the task of VTE prediction, but rather to show that they were not necessary for achieving optimal performance with reduced dimensionality on our data set.

3.3.2 Feature set comparisons

Table 5 lists the overlap between the feature sets of the three presented feature selection methods. The full list of features selected by each method is provided in Appendix A.

Table 5. Overlapping features between feature sets.

Feature selection methods	Features
Expert + Filter + Wrapper*	Site, stage
Filter + Wrapper	Site, stage, antineoplastic-aromatase inhibitors, albumin, creatinine serum, red blood cell count, mean corpuscular hemoglobin concentration (MCHC), mean corpuscular hemoglobin (MCH), protein, mean corpuscular volume (MCV), histopathological type
Filter + Expert	Site, stage, hemoglobin, platelet count, white blood cell count

• * The overlap of only the expert and wrapper feature sets produces the same list of features.

All features deemed clinically relevant were also found to be statistically correlated with the presence of VTE in our filtered feature set. Furthermore, all three feature selection methods selected the cancer site and stage as important features for VTE prediction. While cancer site is a widely used risk factor for VTE, cancer stage is not typically included in currently used scoring systems [19-22]. The clinical team further concurred with the data-driven finding of the importance of clinical staging information.

The overlap of the clinical expert and wrapper feature sets matches the overlap of the clinical expert, filter, and wrapper feature sets and is thus omitted from the table.

3.4 Performance validation on held-out data

The remainder of the results section shows the performance when validating our RF model trained with 500 trees on our held-out data (20% of the original data set).

The ROC curve in Figure 2 illustrates the test performance of the RF model with 500 trees being trained on our entire feature pool in comparison to the ROC curve generated from the Khorana score on our held-out test data set. The model achieves a statistically significant improvement in AUROC of 16.1% compared with the Khorana score. This increase in performance confirms the potential for improving VTE prediction through the inclusion of new risk factors in a machine-learning approach. Next, we validated the 500-tree RF model with each of the previously examined feature subsets.

The ROC curves in Figure 3 show this performance by feature set when run on our held-out data. As in the results in Section 3.3.1, the model trained on the wrapper-selected features did not result in a statistically significant decline in performance compared with the model trained on the entire feature pool. This validates our takeaway that the wrapper feature selection process provided an effective way to reduce the feature space without impacting performance. A full list of DeLong test comparisons for the 500-tree RF models on the held-out data set are provided in Appendix B.

For additional validation, we evaluated the precision-recall curve (PRC) for the 500-tree RF model on each feature set. These results are displayed in Figure 4.

Similar to the ROC results, the PRC curves in Figure 4 show that the models trained on all features and on the wrapper-selected features are the best-performing models and achieve comparable performance.

4 DISCUSSION

In this study, we examined the utility of using machine learning to predict VTE in cancer patients. We accomplished this through a carefully designed set of steps adhering to a typical machine-learning pipeline. First, we selected a feature pool based on the data availability within our patient population. We also set aside 20% of the data in a held-out data set for final model validation. We then performed a number of feature selection methods and trained multiple machine learning classifiers with different hyperparameter configurations to identify a best-performing model for our use case. Finally, we iteratively trained the best-performing model to accumulate a minimum set of required features and thus reduce the complexity of the model without impacting model performance.

The results of this process allow us to draw insight into how a machine learning classifier might offer an improvement in performance over traditionally used clinical VTE risk assessment systems in cancer patients. With these results, we are able to examine our feature pool and identify those features that are most useful in the context of developing an efficient machine learning classifier by comparing the selected features and resulting model performance across multiple unique feature selection methods.

This project was an effort to showcase the improved predictive performance of various ML models over the Khorana score in predicting VTE in cancer patients. We compared the performance of models trained on different feature sets selected by domain experts, statistical methods, and ML techniques. We identified features that were common across these selected feature sets to better understand which features are meaningful in this context.

Our trained classifiers achieved encouraging results on numerous feature subsets. We found that a 500-tree RF model trained using only the features used in the Khorana score achieved a statistically significant 14.6% improvement in AUROC over the standard point-based Khorana score on our held-out test set with an AUROC of 0.769 ± 0.007. Meanwhile, we achieved a peak AUROC of 0.779 ± 0.006 on a held-out data set when training the 500-tree RF model on our entire feature pool. This surpassed the performance of the Khorana score on the same data set by 16.1%. We were additionally able to reduce the number of required features to 15 total (a 48% reduction) without a statistically significant impact on model performance by using a wrapper method to iteratively accumulate features. We also used two model-agnostic feature selection methods—a statistical filtering method and a clinical expert method—which both achieved AUROCs of 0.771 ± 0.007 (mean ± SD) and 0.757 ± 0.004 (mean ± SD) respectively on our held-out data set. All of these results showed statistically significant improvements in performance over that of the Khorana score.

The results in Table 5 depict the overlap between the features selected by our three described feature selection methods. Only cancer site and cancer stage were common across all three feature sets. Cancer site is already a common risk factor considered in current VTE risk stratification systems [19-22]. Based on our experimental results, cancer stage merits inclusion in future VTE prediction systems using an ML approach. Meanwhile, all of the features deemed clinically relevant were also found to be statistically significant in the filtered feature set. Unlike the other two feature sets, the wrapper-selected feature set did not include hemoglobin. However, it did identify three related metrics—MCH, MCHC, and MCV—as essential metrics for VTE prediction. While these metrics are not identical to hemoglobin, they are likely interrelated. Furthermore, since the wrapper method optimizes the feature space based on empirical performance of different feature combinations, an excluded feature is not by necessity unimportant. Instead, an excluded feature may be redundant when compared with the optimal set of features, making its inclusion unnecessary for improving prediction performance.

In comparison to the features used in the Khorana score, all but BMI are included in the filtered and clinically relevant feature sets. Furthermore, the cancer site, which is the most heavily weighted risk factor in the Khorana score, was selected in all three feature sets. Interestingly, BMI, which is included in the Khorana score, Vienna CATS, and PROTECHT, was not identified as useful in any of our acquired feature sets [19-21]. Aside from BMI, however, the results of this study suggest that the predictors used in the Khorana score have a relatively high predictive power when used in a machine-learning context. The results also suggest that the stage of the cancer is useful in predicting VTE and should be considered in future machine-learning applications. Because staging information is not always readily available in medical notes, future studies could look to reliably extract this information from free medical text using NLP methods. Since cancer staging can vary over time as new information comes in and is incorporated in the staging determination, this problem is particularly challenging with past efforts achieving only limited success [42, 43]. One approach that may improve this performance without sacrificing too much predictive power in VTE risk assessment could involve reducing the cancer stage to a binary variable that simply indicates the presence or absence of metastasis [44].

While the results of this study are promising, it is important to note that the data set uses a small sample size, especially for certain subgroups, (i.e., only a few pharmacological groups were used in the patient population). Also, the study did not include cancer patients who had radiation therapy. There is increasing evidence implicating radiotherapy in CAT in cancer patients, however accessing data from the radiation therapy information system was not possible for this study. This study dealt with the patient population at only one location, so before we generalize these results across the general population, the findings in this study should be validated in other patient populations. Furthermore, this study takes a time-agnostic approach to identify useful predictors for VTE in cancer patients. Therefore, this approach highlights VTE predictors that may be useful in a machine-learning context but does not yet reflect an implementable clinical scenario. With this being the case, the aim of this study was to effectively identify these useful predictors to provide the groundwork for exploration of this problem in specific clinical scenarios (i.e., at different stages of prediagnosis presentation, establishing diagnosis, and postdiagnosis treatment phases of a patient's cancer management).

The methods used in this study could be generalizable to other clinical conditions, particularly ambulatory settings, where there is moderate to strong increased risk for developing VTE, such as, congestive heart or respiratory failure, hormone replacement and oral contraceptive therapy, antiphospholipid antibody and other thrombophilia syndromes [45]. Even though multiple studies have demonstrated that thromboprophylaxis using anticoagulant treatments such as LMWH can reduce the likelihood of VTE events, due to the need for training the patients and caregivers to administer (parenteral) the LMWH, regular lab monitoring and dose adjustment, as well as the potential for bleeding complications, all of which add to the cost and quality of care, such prophylaxis may not always be feasible and risk-free. There is thus a need for effective VTE risk stratification and decision support systems to ensure that prophylaxis is administered only to high-risk patients.

The project goal was to select the necessary and sufficient features from our available feature pool that would maximize the predictive power of various statistical ML models. It can be a hard decision to initiate prophylaxis against VTE, especially in ambulatory cancer patients where anti-thrombosis prophylaxis can be expensive and cumbersome. Evidence-based decision support is crucial for minimizing risk in this decision process and improving patient outcomes.

At the POC where the decisions are made, ideally, prediction tools and scoring systems should automatically retrieve the required features and inform the clinicians to help make decisions. For ease of use and interpretability, the list of features should be small, but should provide meaningful enough information to supplement the current evidence and clinicians' evaluations. We found cancer staging information to be particularly meaningful as a predictor of VTE as it was selected in all of our feature selection processes. The Khorana score does not include the cancer staging information as often it can be hard to retrieve accurate staging information from clinical notes. Accurate staging information is often established by cancer registrars retrospectively, which may take up to 6 months. Our study emphasizes the importance of cancer staging information as a predictor of VTE in cancer patients and highlights the need for its timely evaluation. Simplifying the cancer stage variable into a binary value indicating whether the cancer is metastatic (stage 4) or non-metastatic could improve the accessibility and real-time accuracy of staging but would require further studies and additional validation.

5 CONCLUSION

Machine learning offers a promising avenue for improving the performance of current VTE prediction scores in cancer patients. A combination of a time-agnostic approach and three unique feature selection methods demonstrates that at least four of the features that are used to calculate the Khorana score can also provide high predictive power to a machine learning classifier. We also observe that cancer stage information is generally more useful than BMI as a predictor in our ML classifiers. Consultation with clinicians reveal a potential reason—BMI can vary as patients lose significant weight due to cancer itself, chemotherapy, and associated anorexia or other adverse effects. Furthermore, with significant improvements in the generated ROC curve, it is clear that a machine learning classifier can make complex deductions that may allow it to outperform currently used VTE risk scores. The results in this study offer a foundation from which future machine-learning approaches to VTE prediction in cancer patients can be built. Future studies should consider the identified relevant variables in the context of a temporal analysis in which machine learning may be used to dynamically assess at all levels how cancer management progress, including medical intervention, over time can alter a patient's risk of developing VTE.

AUTHOR CONTRIBUTIONS

Samir Khan Townsley: Conceptualization (lead); formal analysis (lead); investigation (lead); methodology (lead); visualization (lead); writing—original draft (lead); writing—review and editing (supporting). Debraj Basu: Conceptualization (supporting); investigation (lead); methodology (supporting); validation (supporting); visualization (supporting); writing—original draft (supporting); writing—review and editing (lead). Jayneel Vora: Data curation (supporting); methodology (supporting); software (supporting); writing—review and editing (supporting). Ted Wun: Supervision (supporting). Chen-Nee Chuah: Conceptualization (lead); formal analysis (supporting); funding acquisition (equal); investigation (supporting); methodology (supporting); validation (supporting); visualization (supporting); writing—original draft (supporting); writing—review and editing (supporting). Prabhu R. V. Shankar: Conceptualization (lead); formal analysis (supporting); funding acquisition (equal); investigation (supporting); methodology (supporting); project administration (supporting); validation (supporting); visualization (supporting); writing—original draft (equal); writing—review and editing (lead).

CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest.

ETHICS STATEMENT

Appropriate institutional review board (IRB) review and approval was obtained from the UCDMC IRB, bearing number: UCDMC.

INFORMED CONSENT

Informed consent was obtained from all patients before inclusion.