Assessing computational predictions of the phenotypic effect of cystathionine-beta-synthase variants

2.1 Dataset

The CAGI1 CBS challenge included 51 synthetic single amino acid substitutions within the human CBS (Table 1), while the CAGI2 CBS challenge included 78 single amino acid variants that had been observed in patients with homocystinuria. The experimental data used in CAGI1 and CAGI2 were published after the challenges were closed by Dimster-Denk et al. (2013) and Mayfield et al. (2012), respectively. Only one variant, p.N228K, overlapped between CAGI1 and CAGI2. In addition, four positions (p.H65, p.L154, p.V354, and p.V371) overlap between the two challenges but the amino acid substitutions are different. The variant nomenclature refers to the human CBS cDNA (Refseq NM_000071.2).

The functionality of the variants was tested in an in vivo yeast complementation assay, where the human CBS allele is expressed in yeast and functionally complements the yeast ortholog, CYS4, which was removed from the chromosome. In this assay, growth is dependent upon the level of mutant human CBS function, and growth rates are expressed as a percentage relative to wild-type grown with the same amount of exogenous pyridoxine supplementation. An experimental standard deviation is also available based on 3–4 repeated assays. The assay was performed in the presence of high (400 ng/ml) and low (2 ng/ml) pyridoxine concentrations. For a detailed description of this assay, see Mayfield et al. (2012) and Dimster-Denk et al. (2013). Participants were asked to submit predictions on the effect of the variants on the function of CBS both in high and low-pyridoxine concentrations. The submitted prediction was requested as the percent of growth when compared with wild-type yeast, with a standard deviation. The predictions were then assessed against the percent of growth values actually measured for each substitution in the yeast assay.

2.2 Prediction assessment

The correlation between the predicted effects of the mutations and the actual effects serves as an initially simple but powerful measure to assess the accuracy of the prediction methods. Because the mutation data are not derived from a normal distribution, nonparametric tests were used to assess the methods. Both Spearman's rank correlation and Kendall's Tau correlation (KCC) were used to assess each algorithm's predictions against the observed growth rates. The root-mean-square deviation (RMSD) was also calculated to estimate the difference between experimental and predicted values. To assess the accuracy of the algorithms in a clinical sense, evaluation was also conducted against a binary version of the experimental growth rates. A threshold of 0 was chosen for the binary version and the performance was evaluated in terms of area under the ROC curve (AUC), sensitivity, specificity, accuracy, and positive/negative predictive value (Lever, Krzywinski, & Altman, 2016). For experimental data, the total number of negatives (no growth substitutions) were defined as N =TN+FP and the total number of experimental positives (growth detected) as P =TP+FN. All the performance indices are shown in Table S1. An overall ranking of the methods was defined as a mean ranking of three different measures (KCC, AUC, and RMSD) shown in Table S2. These three measures were chosen since they assess complementary aspects of prediction performance.

2.3 Bootstrapping

To assess the robustness of our comparison of different prediction models, and to identify any possible performance bias generated by outliers, we performed bootstrap simulation on measurements. 10,000 random samples were generated using sampling with replacement of the experimental data. The resulting evaluation metrics (KUC, AUC, and RMSD) were plotted as bar plots with error bars representing one standard deviation below and above the mean value for each metric generated through iterations of bootstrap sampling of data-points.

To estimate the statistical significance of pairwise prediction comparisons, we applied bootstrap simulation (as described above) to the metrics of interest (KCC, AUC, and RMSD). We then performed an all-vs-all comparison recording the p-values, and corrected for multiple hypothesis errors using the Bonferroni method.

2.4 Characterization of easy and hard to predict variants

We examined the mutation effects that were easiest and hardest to predict to determine whether they shared any common features. We first identified these mutations by summing the binary predictions (0 for no growth, > 0 for growth) at low-pyridoxine conditions across all methods for each variant. The variants with the lowest and highest summed scores were individually examined in terms of sequence, solvent-accessibility, and location within the CBS structure.

To determine the likelihood of a variant being benign or deleterious through sequence analysis, scores for amino acid substitutions were taken from the BLOSUM90 substitution matrix (Henikoff & Henikoff, 1992). Scores of − 1, 0, or 1 were classified as moderate substitutions, that is, substitutions with a likelihood of arising by chance in terms of evolution and therefore of unknown effect on CBS function. Scores > 1 were classified as conservative substitutions with a projected benign effect on CBS, while scores < −1 were classified as nonconservative, indicating a potentially deleterious effect on CBS function. Solvent accessible surface area (SASA) was calculated for the human CBS monomer (PDB id 4COO) using GetArea (Fraczkiewicz & Braun, 1998) and when different, dimer SASA results were noted. Secondary structure assignments and analysis were according to PDB id 4COO and visual inspection of the structure.

2.5 Method uniqueness in prediction results

For CAGI2, evaluation of the specific contribution of each prediction to the variance with experimental results was addressed using a multiple linear regression model. First, a multiple linear regression model was built with the best methods from each group. The top method from each group was chosen based on the highest adjusted R² values of every single method, to exclude predictions using modified versions of the same methods. The final methods included in the model were SID#16, 23, 26, 27, 29, 34, 36, and 41. Subsequently, methods were removed one at a time, and the linear regression equation was recalculated. The contribution of each method to the model was estimated from the delta adjusted R² values. SID#25 was excluded from the model as it lacked predictions for 10 substitutions.

3.1 CAGI1 challenge

In the CAGI1 CBS challenge, participants were asked to submit predictions to assess the impact of 51 single amino acid substitutions upon the function of the human CBS enzyme in both high (400 ng/ml) and low (2 ng/ml) pyridoxine concentrations. The function of the variants had been experimentally tested in an in vivo yeast complementation assay (Dimster-Denk et al., 2013). Twenty predictions from 13 groups were submitted to this challenge (Table 2), which were assessed blindly. A summary of each method is described in Supporting Information. Of the 13 participating groups, nine submitted one prediction, two contributed two, one submitted three and one provided four different submissions. Some methods used sequence-only or structure-only information, some employed meta-predictors, and others combined sequence, structural and annotation data. SID#17 (submission identification number) submitted only raw data without predictions and was excluded from the assessment. Most participants (18/19) provided predictions for both high and low-pyridoxine concentration; however, seven predictions did not distinguish between the different cofactor levels. The majority of the predictions did not include standard deviations (13/19), and most of the methods that included estimates of reliability for each prediction appeared to be arbitrary (constant values like σ = 5, 10, and 15; n = 5/7).

3.1.1 Assessment across different performance metrics

No single evaluation measure can capture a method's performance; thus, various measures were used to assess the phenotype prediction programs, including Kendall's Tau correlation coefficient (KCC), precision/recall, accuracy, and root-mean-square deviation (RMSD), inter alia (Table S1). For KCC, most of the prediction methods display statistically significant correlation with experimental data at both pyridoxine concentrations (Figure 1). Methods SID#2, SID#5, and SID#20 showed strong correlation at both high and low cofactor concentrations. SID#7 was the second best at high pyridoxine concentration (τ = 0.50, p = 4.87 x 10⁻⁶); however, it showed little correlation at low cofactor concentration (τ = 0.25, p = .034). For RMSD, SID#5, which is a meta-predictor, was the best and second best at low and high cofactor concentration, respectively. The highest accuracy of 82.4% was achieved by three methods: SID#1, 6, and 20 (Table S1). The first two combined sequence and structure information, while SID#20 is a meta-predictor, integrating several prediction methods. SID#2, 3, 9, and 11 achieved 100% sensitivity, whereas other methods had the highest specificity (94–100%, SID#7 and SID#14) at both cofactor concentrations. The most sensitive models were mostly structure-based. The majority of the methods had good results for PPV, where the values varied from 65% to 100%; however, for NPV, the values displayed a much wider range (0–90%), meaning that the methods are better at predicting benign than deleterious variants.

3.1.3 Easy and hard to predict variants

We examined variants that were the easiest or hardest to predict based on the consensus output of all methods to determine whether they shared any common features (Figure 2b, Figure S7). At low cofactor concentration, there were overall 12% (2/17) deleterious variants and 18% (6/34) benign variants whose effects were predicted incorrectly by more than half of the methods, our definition of consensus.

The deleterious mutations that were easiest to predict at a low cofactor concentration were p.L154P, p.N228K, p.G258R, and p.G457E. The majority of these are nonconservative substitutions and are located within helices in the CBS structure (Table S3). The easiest to predict benign variants (low pyridoxine) were p.K271E, p.V356A, and p.T383S, with moderate conservation scores and were again mostly located within helices. Among the better predicted benign variants, the majority were partially or fully exposed to the solvent.

The hardest to predict deleterious variants at both high and low-pyridoxine concentrations were p.H65L and p.F385Q. p.H65 is located in the H1–H2 loop and axially coordinates the iron atom on one side of the heme plane, with C52 on the other, and mutation of either of these residues results in low-catalytic activity (Ojha, Wu, LoBrutto, & Banerjee, 2002). Interestingly, the functional impact of these variants was not easy to assess from sequence and structure comparisons. Although p.H65 and the sequence flanking it are locally conserved, the heme-binding domain itself (comprising approximately the first 70 N-terminal residues), with the exception of a short 5-residue helix, has no secondary structure elements and does not resemble other heme proteins in either primary sequence or tertiary structure (Kumar et al., 2018). p.F385Q is located in the H17–H18 loop that forms part of the linker connecting the N-terminal catalytic domain with the C-terminal regulatory domain, and lies within an aromatic cluster of residues p.Y381, p.F332, p.F334, p.F385, p.W390, and p.F396 enclosed by salt bridges p.R336-D388, p.K394-E302, and p.K384-E302 connecting helices H12–H14, H17, and H18. Erroneous coordination between aromatic residues can disrupt the extended pi-pi networks formed by aromatic clusters, thereby affecting protein stability and folding (Madhusudan Makwana & Mahalakshmi, 2015). In addition, both these variants involve nonconservative substitutions, and thus could be expected to have a deleterious effect on CBS function.

The hardest to predict benign variants (low pyridoxine) were surprising in that they involved nonconserved substitutions, so they could be expected to disrupt CBS function. In addition, within the structure, some were implicated in functionally relevant regions of CBS, such as the dimer interface (p.L345P) and the active site (p.V118G, adjacent to the PLP-ligating p.K119). All inaccurate consensus predictions of benign variants at low pyridoxine were for variants that confer sensitivity to reduced pyridoxine levels relative to the major allele (Dimster-Denk et al., 2013). The methods that correctly predicted all of these variants (SID#2, SID#3, and SID#9) displayed a broad spread both in features used (sequence, structure, and thermodynamics, respectively) and in overall performance (Table 2). In addition, SID#2 and SID#9 did not distinguish between high and low pyridoxine.

3.2 CAGI2 challenge

In the CAGI2 CBS challenge, 84 single amino acid variants that had been observed in patients with homocystinuria were collected and functionally tested in an in vivo yeast complementation assay (Mayfield et al., 2012). Seventy-eight had experimental values for both pyridoxine concentrations; 6 “hem1 rescue” variants were left out from the assessment due to absent/conflicting data (Table 1). Participants were again asked to submit predictions of the effect of the variants on the function of CBS both in high and low cofactor concentration. This challenge attracted 20 submissions from nine groups (Table 2) that were assessed without knowledge of the identity of the predictors. An overview of the methods is provided in Supporting Information. Four groups submitted one submission each, three groups submitted two each, and one group each contributed four and six predictions, respectively. Four groups participated in both CAGI1 and CAGI2 CBS challenges. As in CAGI1, features used to generate the predictions ranged from sequence- or structure-only information to meta-predictors and methods combining sequence, structural, and functional annotation data. SID#31 was excluded from the assessment due to its constant growth rate prediction of 100 for all substitutions. Almost all groups (17/19) provided distinct values for high/low cofactor concentrations. For this challenge, most submitters also provided standard deviations (13/19). Only one of the methods had arbitrary standard deviation values for all predictions (SID#26, σ = 10). In addition to prediction programs, reference results were obtained by submitting the mutations to the SNAP (SID#50) and SIFT (SID#51) public servers (Bromberg, Yachdav, & Rost, 2008; P. Kumar, Henikoff, & Ng, 2009).

3.2.1 Assessment across different performance metrics

The same assessment measures as in CAGI1 were used in CAGI2 (Methods). Looking at KCC, over half of the predictions were highly significant (Figure 3); however, even the best deviated substantially from experimental values. At both high and low-pyridoxine concentrations, methods SID#16 and SID#26 showed the strongest correlation with the experimental data and had also high AUC values (Figure 3). The latter was also the top predictor in terms of RMSD. In terms of accuracy, a structure-based method (SID#25) had the highest value (72%) at high and low-pyridoxine concentration (Table S2). At both cofactor concentrations, methods SID#16, 26, 34, and 41 had the highest sensitivity (100%). Most of these methods employed integrated sequence and structure information. SID#23 was the top method (high pyridoxine) with a 75% specificity, SID#27 and SID#28 scored 83% (low pyridoxine). SID#23 used structural data, whereas the other two are meta-predictors. In contrast to the CAGI1 CBS challenge, NPV showed higher median values than PPV (65 vs. 57%), implying that the probability of loss of function was slightly better predicted than the probability of having no or minor effects on the phenotype.

3.2.4 Correlation between methods and unique contribution of different methods

To have a better understanding of the strengths and weaknesses of the different methods, we investigated the correlation between their predictions (Figure 4). Correlation heatmaps for high and low-pyridoxine concentration had negligible differences. The strongest predictor for method correlation appeared to be the relation to a single group (Table 2). For example, SID#27–28, SID#33&41, and all (SID#19–22) except one method (SID#23) were highly correlated among each other. However, SID#29–32 and 35–36 had higher correlations with other methods than among their own group. SID#29–32 were the only predictors that used simply two states (growth rates of 0 or 100). Interestingly, two of the best ranking methods (SID#16 and SID#26) were strongly correlated. In addition, SID# 27, 28, and 34 showed a strong correlation with the top prediction methods, although were based on different features.

Baseline methods SNAP (SID#50) and SIFT (SID#51) showed strong correlation with the best performing predictions, which is partly expected as SID#16 was based on a version of the SNAP algorithm.

To assess the specific contribution of each method to the variance with experimental results in CAGI2, we applied a multiple linear regression model as described in Methods above. For high pyridoxine concentration, this revealed SID#16 and SID#36 as the most significant contributors (Δ adjusted R² values of 0.053 and 0.041, respectively). At the same time, for low-pyridoxine concentration, SID#36 and SID#27 contributed the most (0.054 and 0.053, respectively). SID#36 is based on protein structure, sequence homology, and included functional information, whereas SID#16 combined evolutionary information with structural features, and SID#27 is a meta-predictor (Figure 5).

4.1 Prediction features in relation to performance

In terms of prediction features, different methods performed well in distinct assessment measures. We observed that methods integrating sequence and structural information performed the best overall, ranking first or second (SID#2 in CAGI1 and SID#16 and SID#26 in CAGI2). Methods that used only structural information (SID#9 and 11 in CAGI1 and SID#19–20 in CAGI2) did not perform as well as those combining additional features. However, in terms of individual evaluation metrics, a structure-based method showed the highest accuracy of 72% (SID#25) in CAGI2. In CAGI1, SID#1 and 20 were the best at both cofactor concentrations, reaching accuracy of 82%. The first one combined structure and sequence data while SID#20 is a meta-predictor. In terms of sensitivity, most of the top-performing methods were structure-based in CAGI1 (SID#3, SID#9, and SID#11), while in CAGI2, the most sensitive algorithms combined structure with sequence data (SID#16, SID#26, and SID#34). At the same time, for specificity, almost all of the top methods were meta-predictors in CAGI2 (SID#27–28). These observations suggest that combining different features and methods would yield the best results, as has been indicated previously (Grimm et al., 2015; Tang & Thomas, 2016). Some methods are tailored to predict whether a variant affects the function of the protein in hand and others are optimized to determine whether a variant is pathogenic or benign in the clinical sense (Grimm et al., 2015; Katsonis et al., 2014; Pejaver, Mooney, & Radivojac, 2017).

The importance of integrating information from different sources is reflected in the most inaccurately predicted mutants that tended towards nonconserved substitutions, structural uncertainty, or both. The power of combining structural, sequence, and functional information was visible in CAGI2, where the overall performance of a structure and sequence combined method (SID#35) was improved significantly (by five ranks) with the inclusion of functional annotation data (SID#36). The latter was also the method that uniquely contributed the most predictive power of all methods at low-pyridoxine concentration. Another structure-based method (SID#25) that incorporated functional information (trained on the CAGI1 data set) also performed strongly. Methods trained on HGMD (Human Gene Mutation Database) mutations (SID#35–36) would be expected to perform well (Dong et al., 2015; Ioannidis et al., 2016; Pejaver et al., 2017). Interestingly, however, the best methods in CAGI2 CBS challenge showed variable training data, from no training to training on PMD (the Protein Mutant Database), HGMD, and CAGI1 CBS variant data (Supporting Information).

4.2 Limitations of the challenges

In terms of methodological limitations, most methods were developed to predict pathogenicity in humans or enzyme activity, not yeast growth or the effect of cofactor concentration on growth rate, something that could at least in part explain the difficulties they encountered in identifying the remediable class of variants (Supporting Information). So, while a meaningful distinction could be made in these challenges between growth and no growth under low-pyridoxine conditions, this was not the case for distinguishing between rescue (high pyridoxine) and no rescue (low pyridoxine) variants (Figure S9-12). Similarly, only qualitative comparisons could be made between CAGI1 and CAGI2, since the data sets differed in size and type, and only four groups participated in both challenges. Among these, one group used different versions of their method (SID#1 in CAGI1 and SID#26 in CAGI2), while two others did not make use of the CBS training data.

Another limitation of the assessment involved requesting standard deviation as estimates of reliability from predictors, as opposed to the more commonly employed confidence levels that most prediction methods provide. Consequently, some predictors did not provide these values, chose them arbitrarily, or provided large values with the result that they could not be reasonably used in these assessments.

These challenges also revealed a number of experimental limitations. Yeast CBS lacks the heme domain and is not regulated by AdoMet (Jhee, McPhie, & Miles, 2000), thereby engaging different pathways in the enzyme's regulation and physiological roles. In addition, overexpression can result in nonphysiological effects, including protein aggregation. These differences could help explain some of the inconsistencies observed in the experimental study in which yeast growth phenotypes did not match the clinical data (Mayfield et al., 2012). Although, only three positions from the heme domain were part of these two CBS challenges, with merely one position being problematic for the predictors (Figure 2). In a similar study, several variants identical to the ones used in these experiments resulted in contrasting yeast growth phenotypes (Wei, Wang, Wang, Kruger, & Dunbrack, 2010).

In addition, the clinical assessment of the majority of variants explored in this study has since changed. Of the 78 alleles described in CAGI2 as having been observed in patients with homocystinuria, only 30 are currently classified as pathogenic or likely pathogenic in ClinVar (https://www.ncbi.nlm.nih.gov/clinvar/, accessed March 25, 2019), with an additional 16 annotated as being of uncertain significance or with conflicting interpretations of pathogenicity (Table 1). Eight substitutions (p.P78R, p.K102N, p.D234N, p.R266K, p.V320A, p.T353M, p.V371M, and p.D444N) out of 22 that showed experimental growth rates of ≥ 85% in CAGI2 are currently annotated as pathogenic or likely pathogenic. Most of the participants made accurate predictions for these “benign” variants (Figure 2). It is important to mention here that ClinVar had not yet been launched during the first two CAGI challenges. Also, not all the (likely) pathogenic CBS variants currently present in ClinVar have been collected from clinical testing, some are based on literature with no assertion criteria provided. Ideally, different functional assays should be applied, to increase the confidence in the observed phenotypic effect of the studied variant, because the function of a gene can differ in distinct organisms. Finally, mutations in cis with the ability to either suppress other pathogenic missense mutations or increase the severity of the clinical phenotype continue to be reported (de Franchis, Kraus, Kozich, Sebastio, & Kraus, 1999; Shan, Dunbrack, Christopher, & Kruger, 2001), raising the possibility that the incidence of double mutant alleles may be underestimated in homocystinuric patients.