EXIST: EXamining rIsk of excesS adiposiTy—Machine learning to predict obesity-related complications

2.1 Study design

Electronic health records data of a cohort of patients with overweight and obesity were used to develop and evaluate predictive models for nine long-term clinical outcomes using a) Lasso-Cox proportional hazards models and b) random survival forests (RSF).

2.2 Study cohort

Study participants included adults (age ≥18 years) with body mass index (BMI) between 25 and 80 kg/m² who were being treated in primary care practices affiliated with Mass General Brigham, a large integrated healthcare delivery network in Massachusetts founded by Brigham and Women's Hospital and Massachusetts General Hospital, between 01/01/2000 and 12/31/2019. Patients were excluded from the study if they: a) were older than 80 years old, b) had missing demographic information, or c) were diagnosed with a disease outcome of interest at baseline. The last exclusion criterion was applied on a per-analysis basis (e.g., patients with history of cancer were only excluded from the analysis where incidence of cancer served as the outcome).

2.3 Study measurements

A patient was entered into the study (Index Date) on the first date when they met all of the following criteria: 1) had had at least one primary care encounter (to ensure availability of baseline clinical characteristics, as patients who only receive specialist care at the study institution are less likely to have their medical history comprehensively documented in the EHR); 2) was ≥18 years old; 3) had first of two consecutive BMI measurements ≥25 kg/m²; and 4) on or after Study Start date (01/01/2000). Patients exited the study (Study Exit Date) on the first of the following dates: a) reaching an outcome endpoint (this criterion was specific to the outcome being analyzed); b) death (even when it is not a component of the outcome being analyzed); c) 24 months after the last primary care note (i.e., lost to follow-up) or d) Study End (12/31/2019). Patients were enrolled into the study until 12/31/2018 (to allow at least 12 months of follow-up to assess study outcomes).

Outcomes were obtained from EHR data (both in- and outpatient). Dates of death were obtained from the Social Security Death Master File. The following endpoints were assessed: a) atherosclerotic cardiovascular disease (ASCVD); b) heart failure (HF); c) diabetes mellitus type 2 (T2DM); d) non-alcoholic steatohepatitis (NASH)/ non-alcoholic fatty liver disease (NAFLD); e) sleep apnea; f) cancer (excluding non-melanoma skin cancer); g) degenerative joint disease (DJD); h) knee replacement and i) all-cause mortality. The length of time (days) from the Index Date to each of the clinical events listed above served as study outcomes.

A literature search was performed to identify the candidate predictor variables for study outcomes (Table 1). These predictor variables included demographic characteristics, vital signs and laboratory measurements, past medical history, family history, and history of substance abuse. Predictor variables were also assessed based on EHR data. All variables were assessed at baseline, defined as the most recent record/measurement prior to study entry. For quantitative variables, if no measurements prior to study entry were available, the earliest measurement within 1 month after study entry was used. International classification of diseases codes used to identify candidate predictor variables and study outcomes in the EHR data are provided in Supplemental Table 1.

Missing information for predictor variables was handled using imputation. Imputation was first performed for HbA1c (which was missing for 290,131/67.0% of observations). HbA1c measurements were imputed using random draws from normal distributions based on published data for similar populations, separately for patients with (mean 8.3% and SD 1.7%) versus without (mean 5.6% and SD 0.7%) diabetes.^{18, 19} Imputed HbA1c data were left truncated at 4.0%, which is generally considered to be the lower limit of normal.

Random forest imputation was not used for HbA1c because of the high missingness (67%) of this variable (primarily because it is not typically measured in patients who are not suspected to have diabetes). Previously published reports showed that at this level of missingness random forest imputation performs no better than “strawman imputation” (i.e., imputation of the mean or the median of the non-missing population).²⁰ Under these circumstances, the approach of using data (mean +SD) from the published literature for (separately) patients with versus without diabetes was preferable.

All other missing data were imputed using a random forest algorithm applied to all covariates (using already imputed HbA1c values). MissForest random forest imputation algorithm²¹ was used to impute missing variables on the entire dataset (prior to it being split into training, testing and validation datasets). The MissForest algorithm regresses each variable against all other variables and then predicts missing data for the dependent variable using fitted forest.^{20, 21} The following variables had missing values that were imputed using this approach: a) systolic blood pressure, b) diastolic blood pressure, c) estimated glomerular filtration rate, and d) low-density lipoprotein cholesterol.

2.4 Predictive models

To identify the initial set of variables for the risk prediction model, bivariate analyses of the relationships between each of the candidate variables and each of the outcomes being analyzed were conducted. Candidate variables that were associated with one of the risk prediction model outcomes with a p-value <0.15 were selected for further development of the risk prediction model for that outcome. This approach was taken in part because the ultimate goal of the study was to develop compact predictive models that could be used in patient-facing applications requiring manual data entry and in provider-facing applications drawing data from EHRs, which in some cases may only have complete information on a limited number of variables. Therefore, we prioritized candidate variables likely to have the strongest predictive relationship with obesity-related complications.

The entire dataset was randomly split into 80% training and testing versus 20% that were held out for validation after model development was complete. The training and testing 80% were further split into 70% training and 30% testing (56% and 24% of the entire dataset, respectively) datasets. The overall approach to model training and selection is illustrated in Figure 1.

The training dataset was utilized to build models using Lasso-Cox regression and RSF²² (RSF) methods for variable selection and prediction modeling. Lasso-Cox model regularization selects variables by shrinking the coefficients of less important variables to zero.²³ The optimal penalty parameter lambda for the variable selection was estimated using cross-validation. Variable selection for the RSF model was conducted using the minimal depth approach.²⁴ For every variable used in the growing of the tree, the minimum depth of the variable is the minimum depth of tree nodes split on this variable. Variables used for splitting closer to the root of the tree are applied to larger subsets than those used later in the growing process and are thus expected to have greater predictive power. Variable selection took place in the process of model building. After constructing an RSF model, the variable selection was conducted based on the minimal depth threshold of mean + one standard deviation of minimal depth, and a new RSF model was generated using the selected variables. Both the Lasso Cox regression and RSF models were developed using the 56% of patients included in the training dataset.

The relationship between the variables and the outcome of interest was assessed using a) Lasso-Cox proportional hazards model and b) a machine learning–based method, RSF model. A RSF is an ensemble classification method that determines a consensus prediction by averaging the results of many individual decision trees.²² Lasso-Cox and RSF predictive models were evaluated for each study outcome using multiple replicates of the variable selection and model fitting process (Figure 1). Each replicate included as the first step a random split of the 80% training + test dataset into 56% training dataset and 24% test dataset. Subsequently, both Lasso-Cox and RSF models were developed (including variable selection and model fitting) on the 56% training dataset. These models were then evaluated by calculating Harrell's C-index,²⁵ on the corresponding 24% test dataset. Replicates were stopped once the average prediction error (calculated as 1—Harrell's C-index) stabilized; 100 replicates were used.

The performance of the Lasso-Cox and RSF predictive models was compared using the mean Harrell concordance index (C-index)²⁵ for each outcome across all replicates. This selection process was used to identify the optimal variable selection strategy + modeling approach (Cox regression with Lasso vs. random survival forests) based on both the value of the mean Harrell concordance index across all replicates for the particular outcome as well as on computational resource utilization and interpretability (and ultimately face validity) of the predictive modeling approach. Once the best modeling approach was identified, the final set of variables was selected and coefficients for the model that included these variables were re-estimated using the combined 80% training + testing datasets. Specifically, Lasso-Cox model was computed on the 80% training + testing dataset with cross-validation to select the final set of variables. These variables were then used in the final Cox model that included the evaluation of competing risk of death (except the model for the mortality outcome) as previously described²⁶ and was fitted on the 80% training + testing dataset. Тhe final Harrell C-index was assessed for this final model on the 20% held-out validation dataset.

After the models were developed and evaluated as described above, risk models that included a more limited number of variables appropriate for implementation of provider- and/or patient-facing risk calculators were subsequently selected and evaluated. These (parsimonious) models were developed by using the lasso procedure to identify the order in which variables are added to the model in an incremental fashion to maximize the model fit for any given number of variables (using penalized partial likelihood function) while taking into account the variables' clinical relevance, similar to the previously described approach.^{27, 28} Parsimonious predictive models were developed based on the combined training + testing dataset and evaluated on the held-out validation dataset. The development of the models started with the calculation of the Harrell C-index for each model that had ≤10 variables using the combined training + testing dataset. Body mass index was forced into the parsimonious model if not included using the lasso variable selection procedure. The discriminative ability of models with different number of variables was manually reviewed and a representative model was selected for each study outcome that included the lowest number of variables when the inclusion of additional variables did not lead to significant increases in the model's discriminative ability. A qualitative approach that took into account the trade-off between the increase in the model's discriminative ability (as represented by Harrell's C-index) versus the required increase in the number of variables that would make the predictive model's practical implementation more challenging as well as clinically perceived relevance of the variable being added was used to identify the final set of variables for each parsimonious predictive model. Each parsimonious predictive model developed in this way was then evaluated on the held-out validation dataset.

Analyses were conducted using R (R Foundation for Statistical Computing, Vienna, Austria). The RSF models and random forest imputation were implemented using the randomForestSRC package (R version 3.6.3 and randomForestSRC version 2.9.3 were used).²⁹ Lasso-Cox modeling was implemented using the R package glmnet (R version 4.0.3 and glmnet package version 4.0.2 were used).³⁰

3.1 Study cohort and selection of predictor variables

A total of 448,639 adults with BMI between 25 and 80 kg/m² who were being treated in primary care practices affiliated with Mass General Brigham were identified. After excluding patients who a) were older than 80 or b) did not have information available for demographic characteristics, 433,272 patients were included in the study (Table 2). The median age of the study patients (Table 3) was 48 years and their median BMI 28.9 kg/m²; they were followed for a median of 2030 days (5.6 years). Over this period of time, the incidence of study outcomes ranged from 1.8% for knee replacement to 11.7% for ASCVD (Table 4). Based on the analysis of bivariate relationships between candidate predictor variables and study outcomes, all candidate predictor variables (Table 1) were selected for further analysis.

3.2 Machine learning predictive models

Training of 100 replicates of RSF predictive models for a single outcome took approximately 24 h on 80 central processing unit (CPU) cores of a computational cluster, totaling c. 1920 h (80 days) of computing time. For most outcomes studied, in each 100 replicates, the RSF model selected all available variables for the predictive model. The only exceptions were the predictive models of HF, knee replacement and all-cause mortality, for which the RSF model did not select gestational diabetes mellitus for 37%, 28% and 31% of replicates, respectively. The mean Harrell C-index for RSF predictive models over 100 replicates on the test dataset ranged from 0.702 for NASH/NAFLD to 0.896 for all-cause mortality (Table 5).

3.3 Traditional statistical predictive models

Training of 100 replicates of Lasso-Cox models for a single outcome took approximately 160 s on a single CPU. The mean Harrell C-index for Lasso-Cox predictive models on the test dataset over 100 replicates ranged from 0.694 for NASH/NAFLD to 0.891 for all-cause mortality (Table 5). Across all study outcomes, the average mean Harrell C-index for Lasso-Cox predictive models was lower than the Harrell C-index for the corresponding RSF predictive models by 0.012, ranging from 0.005 for all-cause mortality to 0.020 for T2DM.

Based on its predictive accuracy (slightly lower but comparable to RSF), significantly lower computational requirements and greater interpretability, Lasso-Cox modeling approach was selected for the final predictive model validation. On the training + testing (80% of the overall dataset) dataset, the final Lasso-Cox models were selected from 26 (knee replacement) to 35 (ASCVD) out of 40 available predictor variables (Table 1). Body mass index, age, sex, marital status, HbA1c, hepatitis C, hypertension, smoking status, family history of ASCVD and family history of cancer were included in models for all study outcomes based on Lasso variable selection. On the held-out validation dataset, the Harrell C-index for these Lasso-Cox predictive models, trained on the training + testing dataset, ranged from 0.675 for NASH/NAFLD to 0.849 for all-cause mortality. Body mass index was significantly (p < 0.0001) associated with all study outcomes. Hazard ratios for the relationship between BMI and study outcomes (using one standard deviation of BMI in each respective study population) varied from 1.060 for cancer to 1.602 for sleep apnea (Table 6).

3.4 Parsimonious risk predictive models

Parsimonious Lasso-Cox models included four (cancer and knee replacement) to seven (ASCVD, HF, NASH/NAFLD, DJD and all-cause mortality) variables (Table 7). Body mass index was selected by the lasso procedure as one of the first 10 variables for all models except all-cause mortality and was forced into the all-cause mortality model. Among the rest of the predictor variables, age was included most frequently (eight out of nine predictive models) followed by hypertension (five models). Assessment of model accuracy on the validation (held-out) dataset demonstrated a Harrell C-index ranging from 0.675 for NASH/NAFLD to 0.850 for knee replacement (Table 5).