ESM-scan—A tool to guide amino acid substitutions

1 INTRODUCTION

Proteins play a central role in various aspects of life, operating as fine-tuned molecular assemblies. These systems’ properties define structure and dynamics so that even single alterations in the amino acid sequence can profoundly affect their functionality (Casadio et al., 2011; Tokuriki & Tawfik, 2009; Zhang et al., 2018). Determining the amino acid substitution effects on protein structure, stability and function remain challenging despite continuous advances in structural and functional analyses (Fersht, 1998; Gromiha, 2007; Shoichet et al., 1995). Experimental approaches are often time-consuming and costly, even with high-throughput screening methods.

The earliest computational tools developed for this task: Eris, FoldX, and Rosetta-ddG to name a few, leverage a combination of energy calculations, evolutionary insights, and statistical analyses (Hopf et al., 2017; Kellogg et al., 2010; Ng & Henikoff, 2003; Schymkowitz et al., 2005; Smith & Kortemme, 2011; Yin et al., 2007; Yue et al., 2005). While achieving moderate success rates, these tools require substantial additional knowledge of the methods and the systems on the user's part (Broom et al., 2017; Hsu et al., 2022; Schmitz et al., 2021; Yamaguchi & Saito, 2021). Early machine learning approaches demonstrated good predictive power; however, they tended to overfit training data, limiting their ability to generalize to proteins beyond the original dataset (Avery et al., 2022; Bradford & Westhead, 2004; Rao et al., 2019; Verkuil et al., 2022). Expanding beyond these early approaches, deep learning models harness vast sequence datasets and experimental information, autonomously learning from them unsupervised (Gilmer et al., 2017; Riesselman et al., 2018). These models better capture the individual amino acids' intricate and nonlinear contributions to protein stability and functionality (Ingraham et al., 2019; Sanderson et al., 2023; Mansoor et al., 2023). However, the generalization capacity of deep learning models remains an open topic, with persistent concerns about the risk of overfitting and the inconsistent transfer of learned knowledge to novel proteins beyond the training dataset (Lin et al., 2023; Notin, 2022).

Arguably, the most well-known example of a deep learning model in biosciences is AlphaFold (AF) and its successive improvements (AF2 and AF3), widely acknowledged as a transformative breakthrough in computational protein structural biology (Abramson et al., 2024; Jumper et al., 2021; Senior et al., 2020; Tunyasuvunakool et al., 2021). Tweaks in the original architecture endowed the model with versatility, enabling it to undertake different tasks such as predicting multimeric assemblies, exploring alternative conformations, and evaluating the impact of single amino acid substitutions on protein stability and function with varying levels of accuracy (Bryant et al., 2022; McBride et al., 2022; Pak et al., 2023; Sala et al., 2023; Wayment-Steele et al., 2023). Further developments, like the AlphaMissense model by Cheng et al. (2023), generated a database where single-point variants of the human proteome can be queried for fitness. Another promising research area in deep learning explores language models. These models learn natural language syntax, semantics, and contextual nuances and can generate new meaningful sentences. Drawing an analogy, protein sequences are akin to natural language texts, where amino acids correspond to letters, secondary structures to words, complete sequences to sentences, and assemblies of several proteins to paragraphs (Ferruz et al., 2022; Lin et al., 2023; Wenzel, 2023; Z. Zhang, 2023). Protein-specific language models like ESM, BERT, and their derivatives (Brandes, Ofer, et al., 2022; Chowdhury et al., 2022; Rives et al., 2021), based on the transformer architecture (Vaswani et al., 2017), are capable of learning and extrapolating hidden protein patterns that elude traditional energy-based methods (Chandra et al., 2023). An interesting aspect of these models is their potential use as zero-shot learners, predictors that can run on sets of classes not included in the training data without additional training (Lampert et al., 2009; Larochelle et al., 2008).

The outlined modeling approaches can potentially transform day-to-day research for professionals working in biosciences. However, while some progress has been made, setting up and deploying an ML model is still nontrivial for most users. Moreover, data availability and curation are fundamental in the training and testing phases. Finally, the absence of a computational gold standard method currently hinders the direct transferability of model performances to varied use cases (Avery et al., 2022).

Here, we introduce ESM-Scan, a computational tool leveraging language models from the ESM family to rapidly and efficiently infer the fitness of amino acid substitutions on a given sequence (Brandes, Goldman, et al., 2022; Meier et al., 2021). Our tool is publicly available and requires no set-up from the user, ensuring ease of use. More skilled users can download the source code directly and adapt the tool for more specialized use cases. To illustrate the tool's practical performance, expectations, and limitations, we benchmark its performance against deep mutational scanning datasets also used to benchmark other computational tools described in the literature.

We further provide a demonstration case of the tool using the blue-light-activated diguanylate cyclase from Methylotenera species (MsLadC), a well-characterized in-house model system (Vide et al., 2023). MsLadC acts as a light-regulatable switch alternating between two distinct dimeric conformations: a compact inactive state and an extended active state. In addition to the light sensitivity of the flavin-binding LOV domain and the enzymatic activity of the GGDEF domain, the latter also features a conserved allosteric site for noncompetitive product inhibition (Schirmer, 2016). In MsLadC, this site is also an integral part of an extensive inhibitory interface between sensor and effector domains, governing the protein's inactive, dark state (Vide et al., 2023). Moreover, in other GGDEF domains, this region has been observed to be allosterically coupled to the active site and may also play a role in regulatory protein–protein interactions (Dahlstrom et al., 2016; Hengge, 2021). Originally, we attempted to mitigate the allosteric product inhibition using amino acid substitutions suggested in the literature on related proteins (Teixeira et al., 2021), aiming at increasing the abundance of elongated active conformations for their biophysical characterization. However, these initial variants resulted in low expression yields or nonfunctional proteins. With ESM-Scan, using only the amino acid sequence as input, we could obtain promising alternative single amino acid replacements, less detrimental to protein fitness, allowing variant characterization. We find that ESM-scan scores correlate with functional observations relating to different properties including solubility, cofactor loading, and enzymatic activity, which, coupled with stability, flexibility, and structural integrity, determine said fitness.

Overall, ESM-Scan offers a minimal-overhead and user-friendly resource to the scientific community, guiding the selection of possible amino acid substitutions by predicting their effects and empowering further specialized applications.

2 METHODS

3 RESULTS

Our results show a correlation between ESM scores and the observed impact of amino acid substitutions in proteins, demonstrating accuracies on par with state-of-the-art methods (Broom et al., 2020). Figure 1a illustrates the prediction accuracies across the main datasets, represented by the relative probability density function. In general, the more skewed each distribution is toward the top, the higher the linearity of the data.

3.2 Demonstration case—MsLadC

We aimed to generate protein variants of an in-house model system, MsLadC, that can be expressed and purified, comparable to the wild-type protein but lacking allosteric product inhibition. We focused on Arginine 218 (Figure 2a,b), located within the characteristic inhibitory site (aa-sequence motif RxxD), which is involved in the coordination of c-di-GMP dimers as part of the diguanylate cyclases' feedback inhibition mechanism (Christen et al., 2006; Schirmer, 2016). Based on the promising results obtained in the benchmarking datasets, we applied ESM-Scan to MsLadC (Figure 2a–c). Leveraging insights from the ESM-Scan, we chose to express the MsLadC R218K, R218S, R218N, R218A, R218V, and R218D variants. Despite relatively consistent total expression levels across these variants (Figure S1A), solubility and functionality were significantly affected (Figure S1B and Table S2), highlighting the importance of Arginine 218 in MsLadC, as reflected by the tool's negative scores for substitutions at this position (Figures 2b,c and S2). ESM rankings, purification success, and protein functionality correlate well, with all variants consistently less fit than wild type (Table S2), as anticipated from the diverse roles of R218 and the inhibitory site (see Section 1).

After purification, protein yields for the top-ranking R218K, R218S, and R218N variants were five times lower than wild type, diminishing to 15 times lower yield for R218A. R218V and R218D even lost the ability to bind the FMN cofactor of the blue-light sensing LOV domain of MsLadC, resulting in sensory function loss. In the case of the latter two variants, impairment in sensory function was already visible during cell harvest, as the pellets lacked the typical yellow color. While the protein of interest could still be detected on SDS-PAGE of purified soluble fractions (Figure S1B), the low levels precluded any enzymatic activity assessment. Enzymatic assays for other variants indicated significantly reduced diguanylate cyclase activity compared to the wild type (Table S2). A comparison with the state-of-the-art Rosetta-calculated energies for potential substitutions at position 218 reveals a lack of correlation with the experimental data (Figure 2d).

Our empirical observations from the MsLadC example align with the observed ESM thresholds calculated for dataset 2 (Figure 1b), where values below −6.5 indicate mainly nonviable substitutions (Figure 2d) and values above it, better-performing variants.

4 CONCLUSIONS

Estimating the effect of single amino acid substitution on protein stability and function is still challenging. Traditionally, efforts have focused on stabilizing protein structures (Broom et al., 2017; Kaufmann et al., 2010). Maximal stability, however, does not always translate to optimal functionality (Gianni et al., 2021; McBride et al., 2022; Shoichet et al., 1995). ESM-Scan is a broadly applicable and user-friendly tool designed to predict the functional effects of substitutions based on amino acid sequence. Unlike traditional approaches that focus on the stability of defined structural entities and physical properties, ESM-Scan likely integrates less well-defined properties of functionality; furthermore, it also works for proteins lacking validated structures, for example, multistate sensor proteins or even intrinsically disordered systems. Our analysis, comparing predictions to experimental data from deep mutational scanning experiments, offers valuable insights into the potential and limitations of this tool.

ESM-1v, the original model developed and tested by Meier et al. (2021), demonstrates a certain “understanding” of protein grammar, reporting correlations to experimental results of around 0.5, a value comparable to current state-of-the-art pretrained models (Broom et al., 2020). Our findings point to similar average performances when predicting effects on the stability of monomeric proteins from large datasets. However, ESM-Scan can yield low-accuracy predictions, particularly evident if the test data include significant differences in experimental setup or ΔΔG calculation methods. In practical use cases, variable performance should be anticipated, as depicted in Figure S3. Moreover, even though ESM models have been used to model multimeric complexes, predictions at protein–protein interfaces appear unreliable. Whether this is due to model discrepancies, unfitness, or dataset biases, we cannot make a case for this particular use, and we encourage considering alternative tools (Cagiada et al., 2023; Liu et al., 2021; Sieg & Rarey, 2023).

In tasks where spatial features are pre-eminent, deep learning tools trained on structural information are likely superior predictors, as in the case of protein–protein interactions (Heinzinger et al., 2023; Shu et al., 2023; Wang et al., 2022). Both AlphaFold and ProteinMPNN, a model developed around a message-passing graph neural network, have shown potential in predicting fitness variations upon residue substitution (Brown et al., 2023; Dauparas et al., 2022; Reeves & Kalyaanamoorthy, 2023). Derivatives, like SaProt and ThermoMPNN, extract the parent models' embeddings and, together with structural information derived from PDB structures, train decoder neural networks to predict fairly accurate ΔΔG values (Dieckhaus et al., 2024; Ouyang-Zhang et al., 2023; Su et al., 2023; Umerenkov et al., 2023). Such architectures can model conformational free energy surfaces with reasonable approximation, although challenges persist in efficient structure representation and learning (Jing et al., 2020; Ovchinnikov & Huang, 2021). If atomistic data are available, physics-based tools on the web can provide additional guidance in structural optimization (Goldenzweig et al., 2016; Philipp et al., 2024).

When protein functionality is part of the equation, ESM generally shows state-of-the-art performances. Its inferred values align closely with those provided by co-evolutionary analysis tools and, in some instances, surpass them. This ability could be confined to naturally occurring proteins or domains included during training; future de novo protein functionality assays might allow testing this hypothesis.

In our example scenario, ESM-Scan proved helpful in guiding MsLadC substitution variants of a critical amino acid involved in at least two different functionalities: noncompetitive product inhibition and inactive dark-state assembly formation. Our initial literature-guided rational approaches failed to yield viable protein variants for thorough in vitro characterization. ESM-Scan helped us generate functional MsLadC variants and ESM scores correlated strongly with the experimental findings regarding in vitro protein solubility and functionality, while state-of-the-art bio-physical computational approaches underperformed, highlighting the distinction between stability and more global “functionality”. Furthermore, negative ESM scores generally cluster around functionally important residues and flanking regions (Figure 2b,c), further supporting the notion that scoring relates to a more complex interplay/ensemble of evolution/functionality/stability rather than stability alone as in classical ΔΔG-estimating approaches. Even if all calculated ESM scores are negative, the best-ranked substitution likely outperforms the alternatives. Our illustrative set of variants in the MsLadC system confirmed this observation. While further investigation of R(218)xxD-variants' impact on product inhibition is beyond the scope of this manuscript, the specific activity data presented here underscore the multifaceted role of the inhibitory site, as it strongly influences enzymatic activity also via dynamic coupling with the active site (Christen et al., 2006). While the primary purpose of ESM-Scan is to guide amino acid substitution choices, examination of the overall scoring (Figure S2) may provide insights, not immediately apparent from MSA-derived conservation analyses.

Overall, our findings highlight the importance of considering functionality alongside stability in protein engineering efforts. It is important to emphasize that the primary aim of ESM-Scan is not to enhance or increase protein stability. Other tools, particularly those combined with structural inputs, demonstrably outperform ESM-Scan in this regard. Rather, ESM extracts information encoded in protein language, potentially revealing nonobvious targets for investigation.

By integrating the ESM model's zero-shot predictor into an accessible interface we generated ESM-Scan, an in silico deep mutational scanning tool. Geared towards the casual user, it can operate online with minimal setup, computational resource management, or coding expertise. With fast inference times and minimal overhead, the tool is well-suited for preliminary screenings. Our benchmarking on independent datasets aligns with expected performances. Moreover, we demonstrate the application of ESM-Scan for conducting an in silico deep mutational scan to modulate the functionality of MsLadC. As for any in silico prediction, we want to emphasize again that the user must be mindful when interpreting results, testing the model behavior in each case and adjusting threshold values. For advanced users, the flexibility to adapt ESM-Scan for self-hosted systems offers possibilities for fine-tuning to individual needs and potentially superior results in specific instances.

In essence, the ESM model emerges as a robust method for inferring the impact of amino acid substitutions, especially when evolutionary and functional insights are intertwined. Its inferential capabilities present an exciting avenue for in silico prediction of protein functionality alterations, potentially reducing the need for resource-intensive wet laboratory experiments.

AUTHOR CONTRIBUTIONS

Massimo G. Totaro: Conceptualization; investigation; methodology; validation; visualization; writing – original draft; writing – review and editing; software. Uršula Vide: Investigation; validation; visualization; writing – review and editing. Regina Zausinger: Investigation; writing – review and editing. Andreas Winkler: Conceptualization; investigation; funding acquisition; writing – original draft; validation; writing – review and editing; project administration; supervision; resources; formal analysis. Gustav Oberdorfer: Conceptualization; investigation; funding acquisition; writing – original draft; methodology; validation; writing – review and editing; formal analysis; project administration; supervision; resources.