Objective assessment of the evolutionary action equation for the fitness effect of missense mutations across CAGI-blinded contests

1.1 EA Approach

In order to assess the impact of mutations, we considered a sequence space (Smith, 1970) that mapped onto a fitness landscape (Wright, 1932). There, each mutation should cause a small displacement away from an idealized equilibrium position for the species, defined as an average over the fitness landscape position of all individuals for that species. Let (r₁, r₂, …, r_i, …, r_n)_P be the genotype, γ, of the protein of interest, P, and φ be the fitness phenotype that integrates all the structural, dynamic, and functional attributes that affect the survival and reproduction of the organism. Our hypothesis is that γ and φ are coupled to each other by a continuous and differentiable evolutionary fitness function f, which implicitly accounts for all selection constraints and their variations over time. If so, a small genotype perturbation dγ will change the fitness phenotype by dφ, which will be given by:

(1)

where ∇f is the gradient of f and • denotes the scalar product. For a single amino acid change at sequence position i, from X to Y, the genotype perturbation becomes the magnitude of the substitution (Δr_i,X→Y) and the gradient becomes the partial derivative of f for its i^th component (∂f/∂r_i). Neglecting higher-order terms arising from epistatic interactions with co-occurring mutations (Breen, Kemena, Vlasov, Notredame, & Kondrashov, 2012; Marks et al., 2011), the phenotypic change, or action, of the amino acid substitution becomes:

(2)

This is the EA equation, which states that a missense mutation displaces fitness from its equilibrium position by an amount that is proportional to the evolutionary fitness gradient at that site and the magnitude of the amino acid change. Critically, although the function f is unknown, the terms of expression (Equation 2) may nevertheless be approximated from empirical data on protein evolution.

We approximated the evolutionary fitness gradient ∂f/∂r_i with the relative importance ranks of the ET method (Lichtarge & Wilkins, 2010; Lichtarge et al., 1996; Mihalek et al., 2004). The gradient represents the displacement of the fitness phenotype for an elementary genotype change. We hypothesized that evolution proceeds in infinitesimal steps (Orr, 2005), so any spontaneous amino acid change in protein evolution is an elementary genotype change that adapts fitness in the genetic and environmental context the protein operates (Coyne & Orr, 1998). We also hypothesized that f is continuous and differentiable, so the gradient equals to the difference in fitness phenotype caused by an elementary genotype change. Together, these two hypotheses suggest that the gradient can be measured by quantifying the correlation of amino acid variation and phylogenetic branching, such as the ET algorithm does (Lichtarge et al., 1996). In the extreme cases, invariant sequence positions yield the maximum evolutionary fitness gradient because any genotypic change can displace fitness beyond any homologous protein, whereas positions that vary even between the closest homologous sequences yield the minimum evolutionary fitness gradient.

To measure the magnitude of a substitution (Δr_i,X→Y), we used the odds of observing each substitution in homologous proteins (Henikoff & Henikoff, 1992; Overington et al., 1992). For example, the amino acid alanine is substituted to serine more often than to aspartate, in line with greater biophysical and chemical similarities to the former. However, we found that the substitution odds also depend on the evolutionary gradient of the substituted position. For example, the alanine to valine substitution odds form a bell-shaped distribution as the evolutionary gradient at the mutated position varies from maximum to minimum; those of alanine to threonine begin flat and then tail off, whereas those of alanine to aspartate decay steadily (Katsonis & Lichtarge, 2014). Similarly, differences in the substitution odds were found depending on structural features (Overington et al., 1992). Therefore, we approximated Δr_i,X→Y by substitution odds that depend on the evolutionary importance and on protein structure features of the residue.

2.1 Calculation of the EA

EA scores were calculated according to the public Web server available for nonprofit use at the URL: mammoth.bcm.tmc.edu/uea, where the human protein name and the amino acid substitution may be used as input. Briefly, the EA Δφ of each mutation was the product of the evolutionary gradient ∂f/∂r_i and the perturbation magnitude of the substitution, Δr_i,X→Y. These two terms, ∂f/∂r_i and Δr_i,X→Y, were measured by percentile ranks of ET scores and of amino acid substitution odds, respectively, as described previously (Katsonis & Lichtarge, 2014). All terms, including the EA scores, have been used in the form of percentile ranks, such that high or low scores indicated high or low impact of the genetic variation, respectively. For example, an EA of 68 implied that the impact was higher than 68% of all possible amino acid substitutions in a protein.

2.2 Calculation of other predictors of mutation impact

SIFT predictions were obtained using “SIFT BLink” (http://sift.jcvi.org/), where we provided the GI number of the query protein. Specifically, we used the GI numbers of 4557415 (CBS), 4502749 (p16), 4507785 (SUMO ligase), 32967597 (pyruvate kinase), and 66346698 (NAGLU). The result was a score between 0 (deleterious) and 1 (neutral) for each possible amino acid substitution within the sequence. SIFT scores were treated as the fraction of the remaining protein function over the wild-type function of the protein (0 means 0% and 1 means 100% function).

PolyPhen2 predictions were obtained using the default parameters (HumDiv classifier) of the batch query tab of PolyPhen2 server (http://genetics.bwh.harvard.edu/pph2/), where we provided the NP identifier of the query protein, the protein residue number, and the wild type and substitute amino acids. We used the NP identifiers of NP_000062 (CBS), NP_000068 (p16), NP_003336 (SUMO ligase), NP_870986 (pyruvate kinase), and NP_000254 (NAGLU). We used the “pph2_prob” value as the prediction score, which ranges between 0 (neutral) and 1 (deleterious), to scale it between 0% and 100% loss of the wild-type function of the protein.

2.3 Statistical tests

3.1 SUMO ligase (CAGI 4 - 2016)

A large library of missense mutations in human SUMO ligase was assessed for competitive growth in a high-throughput yeast-based complementation assay, by the laboratory of Professor F. Roth at University of Toronto (Weile et al., in preparation). The challenge was to predict the effect of mutations on ligase activity, experimentally determined by the change in fractional representation of each mutant clone in the competitive yeast growth assay relative to wild-type clones. Specifically, predictors were asked to submit scores between 0 (no growth) and 1 (wild-type growth) for detrimental mutations, and more than 1 for mutants with better than wild-type growth. Data were divided into three subsets of mutants. Subset 1 contained 219 single amino acid variants, each represented by at least three independent barcoded clones and therefore they were assessed with high accuracy (each barcoded clone represented an individual mutant yeast strain). Subset 2 contained 463 additional single amino acid variants, each represented by fewer than three independent barcoded clones. Subset 3 contained 4,427 alleles corresponding to clones containing two or more amino acid variants.

The EA submission (one prediction attempt) treated EA scores as fitness differences. A priori, these differences were assumed to be mostly detrimental, consistent with the nearly neutral theory of molecular evolution (Ohta, 1992). To account for gain-of-function (GOF) variants, however, we then hypothesized that substitute amino acids seen more often than the wild-type amino acid in the homolog sequences alignment could be beneficial, so we assigned negative (“not detrimental”) sign of EA scores for those variants. Since EA scores vary between 0 (wild type) and 100 (loss of function), the activity of SUMO ligase mutants we submitted to CAGI was: submitEA = 1-EA/100. Next, to combine the effect of multiple mutations (M1, M2, …, MN) on the same allele, we multiplied the effect of each mutation, as: submitEA = (1-EAM1/100)·(1-EAM2/100)·…·(1-EAMN/100). When we plotted the average EA scores for bins of 20 variants with similar experimental growth scores (Fig. 1A), we noted that (1) GOF variants had similar EA scores to variants with nearly wild-type activity, (2) variants with experimental growth score between 0 and 1 showed a good correlation with EA scores, and (3) variants with negative experimental growth scores had lower EA impact than variants with zero growth scores, suggesting that these variants may have some activity against the function measured by the assay. On the other hand, when we plotted the average experimental growth scores for decile bins of EA scores (Fig. 1B), we noted linear correlations for each subset, the best of which was for the subset 1 (R² = 0.88) that had the highest experimental growth accuracy. This correlation was consistent to the correlations in E. coli lac repressor (Markiewicz, Kleina, Cruz, Ehret, & Miller, 1994), HIV-1 protease (Loeb et al., 1989), and human p53 (Kato et al., 2003) mutations, that were used to validate the performance of EA (Katsonis & Lichtarge, 2014).

The CAGI assessor for this challenge carefully examined the results by using 18 different assessment metrics to compare the performance of the submissions in each subset. These metrics measured correlations of the experimental growth scores with (1) the original prediction values, (2) the ranks of the predicted values, and (3) a transformation guided by experimental values for the submitted values. Then, as an overall assessment, the CAGI assessor calculated an integrative score for each of these groups of tests and for each data subset, which yielded an overall sum and an overall rank for each method. By this process to define an overall performance score, EA ranked at the top (Fig. 1C). To be clear, the difference between EA and the second best method was small and not necessarily significant. However, all the other methods relied on machine learning and training sets, whereas EA used only the EA equation. Moreover, EA was the only submission with a better overall performance score than a simple conservation-based model developed by the CAGI assessor as a standard of success. To better understand performance, we calculated the Pearson's correlation coefficient and the ROC curves. EA's Pearson's correlation coefficients were only 0.39, 0.38, and 0.26 for the subsets 1, 2, and 3, respectively (Fig. 1D). But these were the best in each data set, including compared with SIFT and PolyPhen2 (which did not participate in the challenge). We note that the area under the ROC curve (AUC) for EA in the three subsets of this challenge was 0.73, 0.72, and 0.70, respectively, for experimental value cutoff of 0.5. These AUC values were also better than the other prediction methods, but they were below the AUC of EA in other datasets (Katsonis & Lichtarge, 2014). To understand this discrepancy, we tested whether the low performance in the ROC metric could be due to experimental uncertainty (Gallion et al., 2017). Indeed, when we restricted the analysis to only account for alleles, in any subset, that had low standard error (SE < 0.05) in the experiments, the AUC rose dramatically, reaching up to AUC of 0.9 (Fig. 1E). We also calculated the AUC of all predictions for nine different thresholds of the growth scores, between 0 and 1. For single mutants, the AUC of most methods increased for low thresholds, suggesting that the computational prediction could separate the partial-function variants from nonfunctional variants better than from the variants with wild-type activity (Fig. 1F). For the multivariant alleles of the subset 3, the cutoff of 0.5 appears to be optimum in separating functional from nonfunctional variants for most submitted predictions.

3.2 Pyruvate kinase (CAGI 4 - 2016)

A large set of amino acid changing mutations of the pyruvate kinase had been assayed in E. coli extracts for their effect on the enzymatic activity and the allosteric regulation of the liver isozyme (L-PYK), by the laboratory of Professor Aron W. Fenton at University of Kansas Medical Center. One subchallenge was to predict the effect of mutations on L-PYK enzyme activity, which was measured as a binary assay result (0, inactive; 1, active). A second subchallenge was to predict the ratios of equilibrium constants for the inhibition of the enzyme by alanine and of the activation of the enzyme by fructose 1,6 bisphosphate. While the first subchallenge is directly relevant to predictions made by EA, addressing the second challenge may require computational analysis beyond the scope of EA. Therefore, here, we focus on predicting the enzymatic activity of L-PYK. Data were split into two experiment subsets: (1) 113 substitutions in nine residue positions, and (2) 430 alanine-scanning mutations.

We used EA to address the enzymatic activity of L-PYK and we submitted one prediction file. EA scores vary between 0 (wild type) and 100 (loss of function), so we treated EA as the probability for a variant to be inactive and we submitted scores calculated as: submitEA = 100-EA. The average activity predicted by EA for active and inactive variants was 30 versus 54 for subset 1 (Mann–Whitney U P value = 7∙10⁻⁶) and 34 versus 59 for subset 2 (Mann–Whitney U P value = 10⁻⁸), respectively (Fig. 2A). When we binned every 20 mutants with similar EA scores, we noted that variants with EA-predicted activity of more than half of the wild-type activity were active in their vast majority, whereas for the rest bins the fraction of active mutants changed almost linearly with the EA prediction (Fig. 2B). This dependence is similar to that of the T4 lysozyme dataset, which we had attributed to sensitivity of the experimental assay (Katsonis & Lichtarge, 2014).

The CAGI assessor of this challenge used the balanced accuracy (BACC) metric to compare the performance of the submitted predictions, for each experimental set. The BACC is given by the average of sensitivity (true-positive rate) and specificity (true-negative rate), which require to set a cutoff for the submitted predictions. The CAGI assessor tested either using as cutoff the value of 0.5, or they calculated the optimum cutoff for each method. For the EA submitted prediction, the value of 0.5 was found to be the optimum cutoff. For each of the two experimental sets, the CAGI assessor found that EA had the top performance according to BACC, even when they optimized the cutoff for the other submitted predictions (Fig. 2C). We reached ourselves the same conclusion when we calculated the AUC of ROC, where EA had AUC of 0.8 and 0.76 in the two subsets, respectively, which were higher than the AUC values of the other submitted predictions as well as of SIFT and PolyPhen2, which did not participate in the challenge (Fig. 2D).

3.3 NAGLU (CAGI 4 - 2016)

The enzymatic activity of NAGLU for 165 missense mutations, which were exclusively found in the ExAC dataset (Lek et al., 2016), was assessed as the percentage of the wild-type NAGLU activity by BioMarin Pharmaceutical, Inc. The challenge was to predict NAGLU activity, submitting scores between 0 (no activity) and 1 (wild-type level of activity), or higher than 1, when the mutation effect was predicted to be detrimental or beneficial. Similar to the pyruvate kinase challenge, we used EA and we submitted one prediction file with scores calculated as: submitEA = 1-EA/100. When we binned every 10 variants with similar enzymatic activity, small enzymatic activities (less than half of the wild type) correlated with the average EA prediction values, but large enzymatic activities (more than half of the wild type) had similar EA scores (Fig. 3A). On the other hand, when we binned every 10 variants with similar EA scores, the average enzymatic activity correlated linearly (R² = 0.90) with EA scores (Fig. 3B).

The CAGI assessor of the NAGLU challenge used three tests to compare the performance of the submissions, the root-mean-square deviation (RMSD), the Pearson product-moment correlation coefficient, and the Spearman's rank correlation coefficient. The assessor did not use the well-established ROC test in their overall rank calculation, because they found that the top five submissions, which included EA, had essentially identical performance with AUC values slightly greater than 0.8. In the overall rank, the assessor included only the best performing submission from each predictor group when a group submitted multiple versions, due to redundancy. According to this overall rank, EA had the third best performance (Fig. 3C). The Pearson's correlation coefficient of EA was 0.54, which was the second highest and better than SIFT and PolyPhen2 (Fig. 3D). We also calculated the AUC of ROC for nine different threshold values of the enzymatic activity between 0 and 1 (Fig. 3E). Most predictions had their maximum AUC at small thresholds, where EA did particularly well with AUC of 0.86 for the threshold of enzymatic activity at 0.3, which was the highest AUC value achieved by any prediction method at any cutoff. We also tested the ROC performance when the analysis was limited to variants with small experimental standard deviations (77 variants had SD below 0.05). Indeed, there was improvement in AUC for almost all predictions, but only for large enzymatic activity thresholds, since variants with low enzymatic activity very often had low standard deviations (Fig. 3F). EA reached a maximum AUC of 0.93 for the threshold of enzymatic activity at 0.9, suggesting strong performance when the experimental measurements were very consistent. The facts that no method had consistently the best AUC of ROC at each threshold and that the relative ranks changed when the analysis was restricted to variants with consistent experimental measurements, support the conclusion of the CAGI assessor that the AUC performance was indistinguishable for the top performing methods.

3.4 P16 (CAGI 3 - 2013)

The ability of 10 p16 variants (CDKN2A gene) to block cell proliferation was tested by Maria Chiara Scaini, at Veneto Institute of Oncology of Padova (Scaini et al., 2014). The p16 variants, like the controls of wild-type p16 (negative) and EGFP vector (positive), were expressed in a p16-null-human osteosarcoma cell line and their proliferation rate was recorded for 9 days. The challenge was to predict the proliferation rate of each p16 mutant cell line relative to the positive control, given that the proliferation rate of the wild-type p16 cells was approximately 50% of the proliferation rate of the positive control cells. We used EA to estimate the impact of p16 mutations, and then we predicted that the proliferation rate would be: submit_EA = 50+EA/2. Although the correlation of the experimental and predicted values was very strong, with a Pearson's r of 0.84 (Fig.  4A), the formula we used to calculate the proliferation rate from the EA scores was subpar. Setting submit_EA = EA would have yielded a much better agreement to the experimentally measured values.

The CAGI assessor of this challenge used four tests to compare the performance of the submitted predictions: Kendall's tau coefficient (τ), RMSD, ROC, and overlap within 10%. The CAGI assessor calculated an overall score from the average rank of these four tests, where EA ranked second out of 22 submissions (Fig. 4B). Of note is that the best submission came from a machine learning method trained with evolutionary and structural features, but the same research group submitted three additional predictions on the same challenge that were trained on different combination of features, and these submissions had intermediate or very poor performance. The EA submission had higher Pearson's correlation coefficient (r = 0.84) than the other submissions and than SIFT and PolyPhen2 (Fig. 4C). Also, the EA submission had perfect ROC, with AUC = 1 in separating the proliferation rate of the variants that had a bimodal distribution (Fig. 4D). The poor performance of SIFT and PolyPhen2 in this challenge was due to predicting maximum impact for almost all of these variants. The best performance of EA on Pearson's coefficient and ROC metrics was consistent between our calculations and the calculations of the CAGI assessor.

3.5 CBS (CAGI 2 - 2011)

The functionality of 84 single amino acid CBS variants, found in homocystinuria patients, was tested in an in vivo yeast complementation assay, by the laboratory of Professor J. Rine, at UC Berkeley. The human CBS clone was expressed and functionally complemented in yeast cells that had the orthologous yeast gene CYS4 removed from the chromosome. In that assay, growth was dependent upon the level of mutant human CBS function, and the rates were expressed as a percentage relative to wild-type (human protein) growth. Two concentrations of pyridoxine, high (400 ng/ml) and low (2 ng/ml), were used. The challenge was to submit predictions on the effect of the variants in the function of CBS in both cofactor concentrations. To address this challenge, we used EA to estimate the loss of CBS activity. At high cofactor concentration, we simply set: submitEA = 100-EA. At low cofactor concentration, we scaled the EA scores to yield lower CBS activities, guided by the test data, such that an EA of 70 will yield 10% CBS activity instead of 30% (linear scaling without changing the extremes, so EA of 0 and 100 will still yield 100% and 0% CBS activity, respectively). Since most CBS variants were found to be experimentally inactive, we binned the variants into those with 0%, 0%–50%, 50%–100%, and more than 100% of the wild-type activity. As expected, the average EA score was higher for the bins of the higher relative growth rate (Fig. 5A). On the other hand, binning every 10 CBS variants by their EA scores yielded strong linear correlations between growth rate and EA (Fig. 5B; R² was 0.87 and 0.93 for high and low cofactor concentration, respectively).

The CAGI assessor of this challenge used nine different tests for each subset of high and low pyridoxine concentration to compare the performance of the submitted predictions, including precision, recall, accuracy, RMSD, Spearman's rank correlation coefficient, F-score, and ROC, among others. Out of 20 submissions, the EA submission had the best performance in nine of the 18 tests, including those of accuracy, RMSD, and F-score, in both datasets. EA was also the best method according to the average rank of all 18 metrics used by the CAGI assessor (Fig. 5C). According to our calculations of the Pearson's correlation coefficients, the EA predictions were the best and the second best method at low and high cofactor concentrations, respectively (Fig. 5D). According to our calculation of the ROC test, EA was the second best method at both cofactor concentrations, with only a marginal difference from the top method (Fig. 5E). When the analysis was restricted to variants with low thresholds of standard deviation, to our surprise, the AUC for almost all predictions dropped, suggesting that lower standard deviations do not imply more accurate experimental measurements in this particular data set (Fig. 5F).