3.1 Data

We used HR data from an urban community mental health centre in the US, which was obtained for the parent study to understand employee turnover mechanisms (NIMH R34MH119411). The HR data were extracted from five different data management systems by the HR, operations and clinical administration departments at the centre. The data were linked to one another by employee IDs, de-identified with research IDs, cleaned and processed for the ML applications by the researchers. Although the majority of data was well maintained in the data management systems, some data were incomplete. The missing data were filled in by HR staff where possible. The historical HR data (2011–2021) for the current study contained information on 621 employees. Among the 621 employees, 56.5% turned over (leavers) during the study period and 43.5% stayed (stayers). The Indiana University Institutional Review Board approved the original study. A waiver of informed consent was applied to the data already collected by the organisation.

We included 14 predictors from the HR data in our ML analyses. Six are categorical: gender (73% female vs. male), race (51% White vs. Black), educational degree (32% Associate degree and below, 34% Bachelors' degree and 34% Masters' degree and above), marital status (26% married vs. single), employee type (77% clinical vs. nonclinical) and exempt status (14% exempt vs. not exempt). Eight are continuous: employee's age, past work years, average hourly wage, average work hours per week, the total number of job training hours, the client information served by the employee (the proportions of male clients and clients with Schizophrenia diagnosis), and average client age. These data were all administratively collected in their routine HR data management. Table 1 shows the descriptive statistics of the included predictors by leavers and stayers.

3.2 Data Preprocessing

Following the standard ML process, continuous predictors were standardised to ensure that their effects would not be influenced by their scale metrics (Hastie, Friedman, and Tibshirani 2001). In addition, because the wage ($) variable was highly skewed (skewness = 5.72, kurtosis = 52.49), we log-transformed it before the standardisation. All categorical data were dummy coded. There was a small amount of missing data on the predictors, ranging from 1% to 7%, except for the client-related variables (47%).

The missing data were imputed using the k-nearest neighbour (KNN) imputation, which is generally considered an effective and robust technique for imputing missing data (Jerez et al. 2010; Kuhn and Johnson 2013; Pereira, Basto, and Silva 2016; Troyanskaya et al. 2001). Simply speaking, KNN first identifies the complete cases similar to a missing case (based on Euclidian distance) and then takes a weighted average (weighted by the distance) of their observations to fill in the missing value (Troyanskaya et al. 2001). Because KNN is a nonparametric imputation method, it does not rely on any distributional assumptions nor assume linear relationships among the variables. Thus, it is unlikely to result in biased imputation or distort the relationships among the variables.

3.3 ML Methods

Below, we briefly introduce six ML methods used in the current study, focusing on conceptual understanding. The ML literature suggests that none of the methods could be universally superior. The relative performance can largely depend on the data set's characteristics (Domladovac 2021; Murugan, Nair, and Kumar 2019); thus, testing the competing methods in our particular study setting is important.

3.4 Cross-Validation

Overfitting (fitting random noise in a specific sample instead of true regularities in the corresponding population) is a common concern in ML methods. To prevent this problem, we used repeated K-fold cross-validation to evaluate the performance of the predictive models (Hastie, Friedman, and Tibshirani 2001). We set K = 10 based on the general practice and size of the data (James et al. 2021). This approach splits the data randomly into 10 equal-sized folds, and each fold is used as the testing data to evaluate the performance of the predictive model trained based on the other ninefolds. This process is repeated ten times, resulting in 100 sets of predictions. The predictions are then aggregated to evaluate the performance of each ML method according to the criteria described below.

3.5 Evaluation Criteria for Prediction Performance

Three criteria were used to evaluate the performance of the prediction models: specificity or true negative rate, sensitivity or true positive rate and area under the receiver operating characteristic curve (AUC). AUC is calculated based on sensitivity and specificity across various thresholds, providing an overall index of how well the cases are classified. An AUC value above 0.8 is generally considered good prediction power (Hosmer, Lemeshow, and Sturdivant 2013).

3.6 Interpretation of ML Results

As mentioned above, additional tools are needed to interpret the results from the examined ML methods. Several graphical and statistical tools were used in this study, including variable importance scores (VIMs), accumulated local effect plots (ALEs), Friedman's H-statistics and Shapley values (SHAPs). VIMs are used to capture the overall impact of each predictor. ALEs are graphical tools for visualising the marginal effects of the predictors. ALEs could also be used to visualise interaction effects. Given the many possible interaction effects (with 14 predictors, there could be 91 two-way interaction effects, not to mention higher-order interactions), we used H-statistics to identify a few impactful two-way interaction effects to investigate. Finally, SHAPs, case-wise statistics, were used for case studies to gain further insights into how leavers and stayers would be different in the potential predictors at the individual case level.

3.7 Variable Importance Scores (VIMs)

VIM is calculated for each predictor, providing a global measure of the impact of each predictor on the prediction (Kuhn 2022). The VIMs are often scaled to facilitate interpretation by dividing the original importance scores by the highest importance scores. Scaled importance scores range between 0% and 100%. For instance, an importance score of 60% means that the importance of the predictor is 60% of that of the most important predictor. There is no absolute criterion to evaluate the scaled importance scores. In practice, researchers often select predictors based on the rank orders of their importance scores (e.g., choose the top 10 scores depending on the number of predictors in the model, see Loh and Zhou 2021).

3.8 Accumulated Local Effect (ALE) Plots

ALEs can be applied to one or two predictors. For one predictor (e.g., X₁), the ALE is a two-dimensional plot showing the marginal relationship between the predictor and a target outcome (e.g., the probability of turnover) averaged across the distributions of all other predictors (Friedman 2001; Molnar, Casalicchio, and Bischl 2020). In other words, it shows how the average prediction could change as a predictor changes. When applied to two predictors (e.g., X₁ and X₂), ALSs could display how the average prediction could change for different combinations of the two predictors across the marginal distributions of the other predictors (e.g., X₃—X_p). Thus, they help visualise two-way interactions.

3.9 Identifying Potential Interaction Effects via Friedman's H-Statistics

Many ML methods (e.g., random forest) can account for potential interaction effects (heterogeneity) without explicitly expressing them in the models. In this case, it requires the use of indices such as Friedman's H-statistics to extract the potential interaction effects (Friedman and Popescu 2008). Briefly speaking, the H-statistics evaluates the degree to which a predictor (e.g., X₁) may interact with any other predictors in the model. There are two types of H statistic values. One represents the proportion of the standard deviation (SD) in prediction from all predictors due to all interaction effects associated with a particular predictor. The other is more specific and can be used to evaluate the strength of a specific two-way interaction (e.g., between X₁ and X₂) relative to the joint impact from both predictors. For convenience, we call the former overall H and the latter specific H. For simplicity, we first used the overall H to identify a predictor for which the interaction effects could be most influential. We then focused on the predictor to explore its potential two-way interactions with any other predictors using the specific H. We used ALEs to portray the top 2 two-way interaction effects associated with the identified predictor.

3.10 Shapley Values (SHAPs)

SHAPs are case-wise statistics (i.e., available for each case), aiming to measure the contribution of each predictor to the prediction relative to the average prediction from all predictors for each individual (Rodríguez-Pérez and Bajorath 2020). SHAPs originate from a well-established coalitional game theory (Hart 1989) by treating predictors as players collaborating to produce the final prediction. As such, SHAPs have many desirable theoretical properties. For example, it provides a fair contribution allocation across the predictors and produces zero values for predictors that cannot predict (Lundberg and Lee 2017; Rodríguez-Pérez and Bajorath 2020). Given that SHAPs are case-wise, they are well suited for case studies which could facilitate our understanding of why a specific employee chose to leave while another decided to stay and what predictors were most influential to their decisions. We present the SHAPs for two extreme cases: one with the highest and one with the lowest probability of leaving.

3.11 Software Implementation

All analyses were conducted in R 4.2.1 (R Core Team 2021). Specifically, the caret package (Kuhn 2022) was used to implement the six ML methods with repeated 10-fold cross-validation and to obtain the variable importance scores. The iml package (Molnar 2018) was used to generate the ALE plots, H-statistics and SHAPs. The R scripts are in the online supplementary materials (https://osf.io/am9nj).

4.1 Predictive Performance

AUC, sensitivity and specificity were used to evaluate the predictive performance of the methods. The average values for these criteria across 100 sets of prediction from the 10-fold repeated cross-validation are presented in Table 2. We used paired t-tests to examine whether the means for each criterion were significantly different for each pair of the methods (six methods, 15 pairs). To avoid inflated type I error rates, the α value for the t-tests was adjusted using the Bonferroni correction (i.e., corrected α = 0.05/15 = 0.0033) (Dunn 1961). Based on the significance of these tests, we ranked the methods for each criterion (the methods that did not differ significantly were grouped together). The rank orders were as follows. For AUC, (RF, GBM) > (LR, EN, NN, SVM); for sensitivity, (RF, GBM) > (NN, SVM) > (LR, EN); and for specificity, all methods had comparable performance (see Table 2).

4.2 Variable Importance Scores (VIMs)

We reported the top five predictors based on their VIMs (see Table 3). As shown in Table 3, past work years appeared to be the most important predictor across all methods. RF and GBM were highly consistent regarding their top five predictors, except for the fifth predictor (i.e., proportion of clients with schizophrenia diagnosis from RF vs. employee age from GBM). A few categorical predictors (i.e., exempt status, educational degree, marital status and employee type) appeared in the top five list from EN, NN or SVM. In total, Table 3 included 10 predictors, with past work years selected by all methods, average weekly work hours and the total training hours selected by four methods, employee age and proportion of clients with schizophrenia diagnosis selected by three methods, average hourly wage, exempt status and educational degree selected by two methods, and marital status and employee type chosen by one method.

4.3 Visualising Marginal Effects via ALEs

To visualise the marginal effects of the selected 10 predictors, we drew an ALE for each predictor in Table 3. If a predictor was included in the top five list of more than one method, we only presented the plot for the method with the highest AUC. For example, we presented the plot for exempt status based on NN instead of EN. More graphs can be found in the online supplementary materials (https://osf.io/am9nj). The ALEs for the 10 predictors are shown in Figure 1.

As shown in Figure 1, the predictive relationships appeared to be nonlinear. However, in general, higher total training hours and the proportion of clients with schizophrenia diagnosis were associated with a higher probability of turnover. On the other hand, more past work years and weekly work hours were associated with a lower probability of turnover. The predictive relationship for hourly wage appeared to be nonmonotonic (e.g., the direction of the relationship changed during the range of the predictor). Although a wage increase in the lower wage range could reduce the probability of turnover, it was associated with an increased probability of turnover in the higher wage range. A similar phenomenon occurred in the relationship between employee age and turnover. An increase in employee age was associated with an increased probability of turnover for younger employees (e.g., <35 years old) but a decreased probability for employees between 35 and 54 years old. Passing around 54 years old, the trend went up again. For the categorical predictors, those with exempt status, were married, held nonclinical positions or had masters' degree or above tended to have a higher predicted probability of turnover.

4.4 Interaction Effects

The ALEs in Figure 1 portray the marginal effects of the predictors but not interactions. Friedman's H-statistics (or H) were used to explore potential interaction effects. We only presented the H for the method that provided the best overall prediction (i.e., RF). Figure 2a displays the overall H for all the predictors. Recall that H values reflect the impact of all possible interaction effects associated with each predictor. The overall H was the highest (about 50%) for past work years, indicating that about 50% of SD in the prediction from all predictors was due to the interaction of past work years with one or more of the other variables. Centring on past work years, we examined all possible two-way interaction effects associated with it using specific H. As shown in Figure 2b, the top 2 two-way interaction effects are the one between past work years and total training hours (H = 37%) and the one between past work years and weekly work hours (H = 22%). These two interaction effects were visualised using ALEs in Figure 3.

As shown in Figure 3, the ALEs for interaction are two-dimensional heatmaps with the x and y axis representing the two predictors that interacted with each other, respectively. The density of the colours reflected the level of predicted outcome for each combination of two predictor values, with lighter colours indicating a higher predicted probability of turnover. Figure 3 shows that newly hired employees are most likely to leave, and those who have worked for a long time are least likely to leave, regardless of the training or weekly work hours. In between, increased training hours could increase the probability of turnover (Figure 3a), while increased weekly work hours could decrease the probability of turnover (see Figure 3b).

4.5 Case Study Using Shapley Values (SHAPs)

Finally, to examine how the leavers and stayers differ at the individual level, we extracted two extreme cases from the sample: a case with the highest probability to leave and a case with the lowest probability to leave. We compared their SHAPs of the predictors based on the RF results (see Figure 4). Note that the lengths of the bars in the figure reflect the magnitudes of the SHAPs, with a higher SHAP indicating a greater contribution of the corresponding predictor to the prediction for the specific case. One can see that the difference in prediction between the two cases was mainly determined by past work years, weekly work hours and the proportion of clients with schizophrenia diagnosis, generally consistent with the predictors' variable importance scores (i.e., VIMs). Taking a closer look at the characteristics of the two cases, we find that although both were clinical employees with nonexempt work status, the leaver case had shorter work history (7 months vs. 3 years), shorter work hours per week (5 vs. 40 h) and worked with more clients with schizophrenia diagnosis (97% vs. 52%). In addition, the leaver case was a younger black male versus an older white female; however, age, race and gender showed minimum influence (i.e., SHAPs) for these two sample cases.

5.1 Limitations and Future Directions

Some limitations of the current study are worth mentioning. First, the ML turnover prediction models were developed in the specific organisational contexts. ML is a data-driven approach that can localise the prediction model. However, this may limit the generalizability of the findings to other mental health settings or organisations. Second, the current sample size was on the small side in the ML literature (Zhao et al. 2019). Although the ML methods are applicable to similar sample sizes like ours (e.g., Quinn, Rycraft, and Schoech 2002; Tzeng, Hsieh, and Lin 2004; Zhao et al. 2019), small sample sizes could compromise the generalisability of the result. Future research is warranted to apply the ML methods to more extensive HR data across multiple mental health centres and to validate the result with external data sets. Third, the result is limited to the predictors included in the study. If more predictors are included, especially those with potentially high impact, then the predictive performance, the rank order of the variable importance score and the relationship patterns could change. Identifying important predictors that are missing in the current HR data and developing practical and efficient ways to integrate them into the HR data collection process would be promising in boosting the predictive power of HR data. Fourth, like other administratively collected data, HR data are not bias free. For instance, if particular subpopulations (e.g., gender, race and age) left the organisation systematically by other structural factors (e.g., insufficient organisational support or discrimination towards minoritised employees), prediction models trained by the historical data can be biased. Finally, the current study predicted turnover probability within the first year and considered only averaged effects across individuals. With more data, future research could be conducted to account for individual changes of turnover probability over time and examine time-varying and time-constant predictors for individual changes. This will require implementing ML methods that are capable of handling longitudinal categorical data with random effects (Cascarano et al. 2023).