Variation Interpretation Predictors: Principles, Types, Performance, and Choice

Introduction

Next-generation sequencing (NGS) methods produce a wealth of detailed information, including large variation datasets. Every human genome contains about 3 million single nucleotide substitutions in comparison with a reference genome, but only a few of those are disease-related. In addition, there are other more complex variations, fewer in number but challenging to interpret. The 1000 Genomes project estimated that each genome codes for about 11,000 amino acid substitutions, approximately 12,500 synonymous substitutions, and hundreds of insertions and deletions [Abecasis et al., 2010].

DNA sequencing has been automated very effectively and thus is fast and inexpensive, especially in the case of resequencing the human genome. The $1,000 genome sequence is already a reality. Sequencing is widely used for diagnostic purposes, but is often done for individual genes. As the price of exome and complete genome sequencing is falling below the cost of sequencing one or a few specific genes, there is increased interest in using genome sequencing in health care.

The fall in sequencing costs has exposed the major deficits in our understanding of the pathogenic significance of most variants. The situation reduces clinical utility and greatly affects the potential for disease treatment and prevention. The problematic lack of knowledge arises because large numbers of the identified variations are novel, or without proper annotation, and knowledge about disease association is missing. Data analysis and variation interpretation are the most time-consuming steps in sequencing projects. Whole-genome or whole-exome sequencing will provide valuable new information about an individual if this information could be utilized and interpreted in a reliable and meaningful fashion. The genome-wide interpretation of variants is only feasible with computational prediction methods. The bottleneck in the utilization of sequence information has shifted from obtaining the sequences to understanding and interpreting them.

Several groups are working on developing guidelines for how to utilize sequencing information for clinical purposes. Recommendations by both the European Society of Human Genetics (ESHG) and the American College of Medical Genetics (ACMG) include the use of prediction methods [Matthijs et al., 2015; Richards et al., 2015]. For certain diseases, sequencing is the only method for differential diagnosis. Diagnosis and prognosis in many diseases is highly dependent on sequence information. Numerous computational tools have been developed for the analysis, prioritization, and interpretation of variations and their effects. End users of these tools are often puzzled by not knowing which method to choose, which ones are the most reliable, and how to choose the most suitable applications. This paper aims at describing what types of predictions are currently possible, how the tools can be compared, and their performance be assessed, as well as some principles that are necessary to understand and to be able to evaluate methods. Finally, tips are provided for how to choose the most suitable computational methods and what we can expect to have in the future. Although computational approaches are very helpful and vital for variation interpretation, one has to bear in mind that these methods alone cannot define whether a variant is pathogenic or not, and other types of information (such as patient clinical information, family history, etc.) are also needed for making such decisions. The type(s) of additional information varies for different diseases.

Several organizations are working toward providing and promoting reliable variation information. The Human Genome Variation Society (HGVS, http://www.hgvs.org/) has been the scientific society for the variation database and interpretation community, variation nomenclature being one of their major achievements [den Dunnen and Antonarakis, 2001]. The Human Variome Project (HVP, http://www.humanvariomeproject.org/) “is working to ensure that all information on genetic variation and its effect on human health can be collected, curated, interpreted, and shared freely and openly.” Recently, the Global Alliance for Genomics and Health (GA4GH, http://genomicsandhealth.org/) was founded to enable sharing of genomic and clinical data.

Variations can have their effects on many levels. The most studied are amino acid substitutions; however, also other types of variations, such as insertions and deletions, are important. Amino acid substitutions can have numerous effects and the mechanisms behind them are diverse [Steward et al., 2003; Thusberg and Vihinen, 2009; Vihinen, 2015c]. They may disrupt or block critical sites for protein function, such as catalytic residues or ligand-binding pockets. They can lead to alterations in the protein structural properties, causing, for example, abnormal folding, structural instability, or aggregation of the protein. Even small changes in the size or chemical or physical nature of an amino acid side chain can be critical. Amino acid substitutions may affect protein posttranslational modifications by generating or deleting such sites, for example, for phosphorylation or glycosylation, or protease cleavage, or altering protein localization targeting signals. DNA and RNA variations often affect important functional sites such as binding sites, for example, for transcription factors or microRNAs or in the case of RNA, the structure of the molecule. For detailed systematic organization of the variation types and effects, see the Variation Ontology (VariO) [Vihinen, 2014b, 2015c].

Most studies are concentrated on loss-of-function variations; however, there are also gain-of-function alterations due to irregular or altered binding of ligands or loss of specificity. Many proteins are robust and can tolerate alterations and insertions to numerous sites without any or just a minor effect on protein function [Poussu et al., 2004]; on the other hand, there are sites that do not tolerate any variations. Recently, some computational studies have been published in which all possible amino acid substitutions or those caused by single-nucleotide changes were analyzed. These include kinase domain variants in Bruton tyrosine kinase (BTK) [Väliaho et al., 2015], variants in four mismatch repair (MMR) proteins [Niroula and Vihinen, 2015a], and predicted protein solubility affecting variations in human interleukin-1 β [Yang et al., 2016]. In addition, similar studies have been performed for mitochondrial tRNA molecules [Kondrashov, 2005; Niroula and Vihinen, 2016]. These studies predicted the effects of all possible amino acid/nucleotide substitutions in the proteins/RNAs and provided important information about the types of variations and their distribution along the molecule sequence and structure highlighting both structural and functional aspects. The frequency of predicted harmful variants varies greatly. Depending on the location and the type of variations, the outcomes are different. The BTK kinase domain has the highest published ratio of harmful variants, with 67% of the studied cases [Väliaho et al., 2015]. In MMR proteins, the proportions of harmful amino acid substitutions varied from 14.6% (in PMS2) to 40.4% (in MSH2) [Niroula and Vihinen, 2015a]. There are only a few experimental studies for massive amounts of amino acid substitutions in single proteins. The ratios of harmful variations vary widely in them, see Väliaho et al. [2015].

Several reviews have been published about prediction methods and their use [Thusberg and Vihinen, 2009; Cline and Karchin, 2011; Capriotti et al., 2012; Peterson et al., 2013], but none of them have covered the full scope of the discussion as done here.

Variation Databases

When investigating the effects of variations, one has to find out whether the identified alteration has been previously found in patients or in healthy individuals. If a variant is frequent in healthy individuals in populations, possibly in several ethnic groups, it is likely not harmful, although there are some exceptions. Central variation databases, such as the Human Gene Mutation Database (HGMD) [Stenson et al., 2014], Online Mendelian Inheritance in Man (OMIM) [Hamosh et al., 2005], and ClinVar [Landrum et al., 2014], include variants for many human genes (Table 1). The UniProtKB/SwissProt [Uniprot Consortium, 2015] database contains manually annotated protein entries with some variants [Yip et al., 2008] available at the SwissVar service (http://swissvar.expasy.org/). For a comprehensive review of the current state and future challenges of genotype–phenotype databases, see Brookes and Robinson [2015].

Locus-specific variation databases (LSDBs) list variants in specific genes/diseases and are typically manually annotated. Many LSDBs contain also other information, sometimes even very detailed clinical data. LSDBs exist at the Leiden Open Variation Database (LOVD) system for all human genes [Fokkema et al., 2011]. Many of them are still empty and without curators. One of the problems is that there can be more than one database for some genes and diseases and they can contain overlapping data [Mitropoulou et al., 2010]. LSDBs are usually the primary and most trusted variation information sources as they are curated and maintained by experts in the genes and diseases. The currently existing LSDBs are listed at The Human Genome Variation Society (HGVS) Website (http://www.hgvs.org/locus-specific-mutation-databases), the LOVD site (http://grenada.lumc.nl/LSDB_list/lsdbs), the GEN2PHEN server (http://www.gen2phen.org/data/lsdbs) [Thorisson et al., 2009], and at the Web Analysis of the Variome (http://bioinformatics.ua.pt/WAVe/) [Lopes et al., 2011]. Other large collections of LSDBs include IDbases [Piirilä et al., 2006], and those maintained at Universal Mutation Databases (UMD) platform [Beroud et al., 2000].

Several guidelines and recommendations have been published for LSDBs including how to establish a database [Vihinen et al., 2012], their curation [Celli et al., 2012], overall contents [Kohonen-Corish et al., 2010], ethics [Cotton et al., 2005; Povey et al., 2010], data collection [Cotton et al., 2007; Cotton et al., 2009], somatic variations [Olivier et al., 2009], interpretation and reporting of variants [Plon et al., 2008; Richards et al., 2015], data sharing [den Dunnen et al., 2009], and nomenclature [den Dunnen and Antonarakis, 2001; Taschner and den Dunnen, 2011]. Beside variant descriptions, LSDBs can also contain clinically relevant information, including where and by whom the variant was found (contact information, publication details), how it was detected (screening methodology), and whether it influences the function of the gene or its product. Some LSDBs contain details of the phenotype of the patient.

The dbSNP database [Sherry et al., 2001] is a very large database for short variants and contains data from The 1000 Genomes Project (http://www.1000genomes.org/). Note that dbSNP contains both disease-causing and benign variants. The NHLBI Exome Sequencing Project (ESP) Exome Variant Server (EVS, http://evs.gs.washington.edu/EVS/) provides exome data from 200,000 individuals from African-American and European-American populations [Fu et al., 2013]. Individuals with specific traits related to blood, heart diseases, and lung diseases as well as controls are included in the EVS project. The Exome Aggregation Consortium (ExAC, http://exac.broadinstitute.org) has data from 60,706 unrelated individuals and the Allele Frequency Community (AFC, http://www.allelefrequencycommunity.org/) currently contains data for about 100,000 exomes/genomes. Some databases annotate human variation data with phenotype variations and protein structural and functional information such as KMDB/MutationView (http://mutview.dmb.med.keio.ac.jp/MutationView/jsp/index.jsp). The University of California, Santa Cruz (UCSC) Genome Browser [Kent et al., 2002], the National Center for Biotechnology Information (NCBI) Map Viewer [Wheeler et al., 2003], the Ensembl Genome Browser [Stalker et al., 2004], and others provide information about genes, their products, and sequence variants. The Encyclopedia of DNA Elements (ENCODE) project [Sloan et al., 2016] aims at identifying and mapping functional and regulatory elements in the human genome also outside protein coding regions. PhenCode [Giardine et al., 2007] connects human phenotype and clinical data in LSDBs with genomic data from the ENCODE project and other resources in the UCSC Genome Browser.

Additional types of variation databases include national and ethnic databases [Patrinos, 2006] such as ETHNOS [van Baal et al., 2010], variation frequency databases including FINDbase [Papadopoulos et al., 2014], chromosomal structural variation databases such as The Database of Genomic Variants (DGV) [MacDonald et al., 2014], Database of Genomic Variants archive (DGVa, http://www.ebi.ac.uk/dgva), dbVar (http://www.ncbi.nlm.nih.gov/dbvar) [Lappalainen et al., 2013], InvFEST database of inversions [Martinez-Fundichely et al., 2014], Mitelman Database of chromosome aberrations and gene fusions (http://cgap.nci.nih.gov/Chromosomes/Mitelman), and European Cytogeneticists Association Register of Unbalanced Chromosome Aberrations (ECARUCA) database containing cytogenetic and clinical information of rare chromosomal disorders including microdeletions and microduplications [Feenstra et al., 2006] (Table 1). Databases dedicated to certain types of variations or to an effect or mechanism are available, for example, in the ProTherm database for protein stability affecting variations [Kumar et al., 2006], and The Cancer Genome Atlas (TCGA, http://cancergenome.nih.gov/) and the Catalogue of Somatic Mutations in Cancer (COSMIC, http://cancer.sanger.ac.uk/cosmic) [Forbes et al., 2011] databases for genetic variations in cancer (Table 1).

From the Cafe Variome [Lancaster et al., 2015], one can find whether a variant has been previously published, but without revealing further information. For more information about the variant, one needs to contact the relevant data owner/provider. One additional type of databases is highly relevant for variation effect predictions, namely, benchmark databases. VariBench [Nair and Vihinen, 2013] and VariSNP [Schaafsma and Vihinen, 2015] collect benchmark variation datasets from different sources and distribute datasets that provide gold standards for method developers and assessors.

It is recommended to use systematics in variation databases whenever possible, including those for phenotype such as the Human Phenotype Ontology (HPO) [Robinson et al., 2008] and the Variation Ontology (VariO) [Vihinen, 2014b] for a systematic description of effects, mechanisms, and consequences of variants. The usage of these systematic approaches facilitates efficient searches, data integration, as well as the development of computational systems for data analysis and interpretation. Several de facto standards are useful for variation databases. These include systematic gene names available from the HUGO Gene Nomenclature Committee (HGNC) [Gray et al., 2015]. Reference sequences have to be specified including version numbers, unless Locus Reference Genomic (LRG) [Dalgleish et al., 2010] entries are used. LRGs are recommended for human sequences as they are stable and allow unambiguous mapping of positions. For naming variations, the HGVS nomenclature [den Dunnen and Antonarakis, 2001] should be used. It is widely used in the literature and also by several computer programs including those interpreting variations. The Mutalyzer tool can generate systematic names automatically and performs a number of consistency checks [Wildeman et al., 2008].

The HVP has released quality assessment criteria to evaluate the quality of genetic variation databases and to stimulate database curators to make improvements where they are needed (http://www.humanvariomeproject.org/finish/19/255.html) [Vihinen et al., 2016]. Once the quality scheme is implemented, the quality information will allow users to check how reliable, up-to-date, and user-friendly a certain database is.

Principles

It is important for users of any method, whether utilized in laboratories or run with computers, to know how they work. Only by doing so can one evaluate the significance of the observed findings. Prediction methods can be rather complex and complicated, especially when considering the details; however, the general principles of these methods can be understood by any scientist. Bioinformaticians seek to find the best solutions for their methods. This means that novel approaches are implemented to improve the performance of the methods and new methods are released frequently.

Machine Learning

A majority of the recent methods in the variation interpretation field are based on machine learning (ML). ML is a form of artificial intelligence where a computer system can learn from given data (Fig. 1). Among the different types of ML techniques, supervised ML classification is widely used for variation interpretation. In supervised ML classification, a computer model is trained to distinguish between two or more classes using given examples. The most common ML algorithms include neural networks, support vector machines, and random forests. The ML methods are highly dependent on the quality of data used for their training. Typically, the methods have been binary, that is, providing two classes of predictions. Some of the methods have more than two classes as variation pathogenicity is far from the binary, black and white, pathogenic or not dichotomy. Binary classifiers can still be useful for many applications. Some methods do not categorize the variations into classes but provide continuous scores.

ML methods are widely used to tackle complex phenomena, which would be otherwise difficult to handle. For a more detailed discussion, see Vihinen [2012], for example. Variations can have large numbers of effects that would be difficult to take into account in other types of predictors. An important aspect of ML method development is the choice of the approach. Among the numerous approaches, there is not one superior architecture. The quality of a predictor depends on how the training is done, which features are used to explain the phenomenon, and how the method is optimized.

Features are used to describe the characteristics of the investigated phenomenon. If several features are available, it is important to choose those that best capture the phenomenon (Fig. 1). This is partly due to the curse of dimensionality, which means that much more data are needed when the number of features increases. In fact, the volume of the feature space grows exponentially with the dimensionality such that the data become sparse and insufficient to adequately describe the feature space. Another problem is overfitting, which means that the algorithm describes noise or random features in the training dataset instead of the real phenomenon due to sparse data, the complexity of the model, an excessive learning procedure or any combination of them. Overfitting leads to a decreased performance on real cases.

ML methods are trained to detect differences in datasets. For that purpose, a good quality training dataset is required. The dataset should represent the space of possible cases [Nair and Vihinen, 2013]. This space is huge for genetic variations as there are so many different effects and underlying mechanisms. ML predictors are trained with known positive and negative instances in an approach called supervised learning that leads to reorganization of the system—learning (Fig. 1). Once the method has learned to distinguish between the given examples, it can be used to classify unknown cases. The training set should contain about equal numbers of cases in each class. An imbalance in the numbers of cases in the classes can reduce the performance of the method [Wei and Dunbrack, 2013].

The same data cannot be used both for method training and testing (Fig. 1). The trick is to partition the dataset. This can be done in different ways, cross-validation being the most popular. The dataset is divided into disjoint partitions, one of which is used for testing and the others for training. This is repeated until all the partitions have been used as test set. The average performance measures computed from the splits are used to describe the overall prediction performance. The prediction performance would be biased if the algorithm parameters and feature sets are optimized using the same or a part of the data. Therefore, the use of an independent test data set is highly recommended. The data in this set should not be used during any of the previous steps of method development (Fig. 1).

Types of Tools

Before starting variation interpretation, one has to assume that the sequencing-related steps have been professionally and properly performed. Steps such as alignment, variation calling, and assessment of read quality are essential prerequisites. Variation interpretation cannot correct mistakes made in earlier steps; therefore, it is essential that the users of these tools are aware of the quality of the detected variants. In this respect, issues such as coverage and sequencing technologies used are very important; however, these are outside the scope of this article. Bear in mind that the interpretations of variation callers do not always agree [O'Rawe et al., 2013].

The effects of many deleterious variants are straightforward to explain, such as large deletions, protein truncations, amphigoric amino acid insertions and deletions that change the sequence after the variation site, and some other types. Usually, the most difficult ones to interpret are minor alterations, most often single-nucleotide substitutions and consequent amino acid changes. The effects on proteins are more difficult to study experimentally than those on DNA or RNA. Hybridization and sequence complementarity based methods work (almost) equally well for all nucleotide sequences, whereas a dedicated analysis protocol is needed for every individual protein and quite often even for individual variants, see Storz and Zera [2011], Yates and Sternberg [2013], and Perniola and Musco [2014], for example. Experimental methods are the first choice to study the detailed effects of variants. Such methods are often expensive and time-consuming and cannot be performed in the scale that exome and complete genome sequencing methods produce variants. Computational methods are thus useful for prioritizing variants for experimental studies [Thusberg and Vihinen, 2009; Zhang et al., 2012; Kucukkal et al., 2014]. Several computational tools have been developed for variation prioritization. The methods vary widely in the principle, implementation, and application.

Prediction of the Effect or Mechanism of a Variant

Tolerance predictors detect variants that can be harmful but cannot explain what is wrong with them and in which way. Variations can have various effects, consequences, and mechanisms (Fig. 2); for those in amino acids, see Vihinen [2015c]. Variation effect analyses have to be done separately as there is no tool to combine the different predictions and interpret their results together. As the methods have widely varying speeds, it is beneficial to run a fast tolerance predictor first and then use tools to predict the effects and mechanisms for the potential harmful variants. This is also warranted by the fact that method and effect-specific tools typically have higher error rates than the best tolerance tools and they can be substantially slower to run. Therefore, filtering variants by tolerance predictors could improve their performance.

For some of the effects, including protein stability, aggregation, and solubility, there are several tools, some with tested performance. The biggest problem for the end-user with many of these predictors is that the method performance has often been presented in a somewhat biased or selective way and without comparison to related tools. Thus, it may be impossible to interpret the quality of the obtained predictions. The situation is improving due to the use of benchmark databases.

Protein stability predictions

Stability affects function, activity, and regulation of biomolecules. Incorrect folding and decreased stability are the major consequences of pathogenic amino acid substitutions [Wang and Moult, 2001; Ferrer-Costa et al., 2002; Stefl et al., 2013; Peng and Alexov, 2016]. Substitutions can cause, for example, a reduction in the hydrophobic area, over packing, backbone strain, loss of electrostatic interactions, and thereby affecting protein stability. Alterations to intramolecular interactions affect the free energy difference (ΔΔG) between the folded and unfolded states of proteins.

Several methods predict stability effects by comparing ΔΔG between a wild-type protein and its variants. Some of these methods rely on energy functions, whereas others apply ML approaches. Methods utilizing energy functions can be subdivided to physical potential approaches, statistical potential approaches, and empirical potential energy approaches [Guerois et al., 2002; Parthiban et al., 2006]. All the ML-based methods are trained with data from ProTherm database [Kumar et al., 2006]. They use features derived from tertiary protein structure, evolutionary conservation, sequence environment, network topology, and properties of amino acid residues to describe each variation [Capriotti et al., 2005; Cheng et al., 2006; Capriotti et al., 2008; Teng et al., 2010; Chen et al., 2013; Yang et al., 2013; Folkman et al., 2014; Giollo et al., 2014; Pires et al., 2014; Fariselli et al., 2015]. It is evident that different features are important for tolerance and stability predictors. Some methods combine both ML algorithms as well as energy functions [Dehouck et al., 2009; Masso and Vaisman, 2010; Laimer et al., 2015]. While most of the methods predict the stability changes for individual variations, some predict the combined effect of variation pairs [Huang and Gromiha, 2009].

Protein localization predictions

To function properly, a protein must be translocated to the appropriate cellular compartment. Proteins are directed to the locations by short targeting signal peptide sequences. Substitutions in signal peptides can disrupt or alter the signal and prevent localization or redirect the produced protein to a wrong cellular part. When a protein is not transported to the correct subcellular location, central reactions can be inactivated or signaling pathways are misregulated. An active mislocalized protein is likely to have harmful effects by exerting its function in the wrong environment.

Despite the fact that numerous predictors are available for predicting the protein subcellular localization, they do not predict the impact of variations on protein localization. Using WoLF PSORT [Horton et al., 2007] and a pipeline of several tools [Emanuelsson et al., 2007], disease-causing variations were studied in relation to protein localization [Laurila and Vihinen, 2009]. Among the 22,416 disease-causing variations tested, hundreds of variations are likely to change protein localization. PROlocalizer was later developed to predict the cellular localization as well as the impact of variations on localization [Laurila and Vihinen, 2011].

Splice-site effect predictors

Splicing of nascent premessenger RNA (pre-mRNA) is an essential step in mRNA maturation. Alternative splicing increases mRNA and protein diversity by generating multiple mRNA products from the same pre-mRNA. Hence, one gene can code for multiple protein products. At least about 95% of human genes have more than one transcript [Pan et al., 2008]; however, many of them are extremely rare and their biological significance has not been established. Among the variations in the HGMD, about 10% of the disease-causing variations occur within splice sites [Krawczak et al., 2007], which is likely to be an underestimation since the exonic variations outside the splice sites and variants at deep intronic sites could potentially disrupt splicing [Cartegni et al., 2002; Hunt et al., 2014]. Among the primary immunodeficiency-causing variants in IDbases, 15% are splice-site variants [Piirilä et al., 2006]. Exonic RNA substitutions are classified as missense, nonsense, or silent variations and their role in splicing is not generally analyzed. Several methods have been developed to predict splice sites [Hebsgaard et al., 1996; Pertea et al., 2001; Yeo and Burge, 2004] and splicing regulatory sites [Fairbrother et al., 2002; Wang et al., 2004; Goren et al., 2006; Smith et al., 2006]. There are also some tools to predict the effects of variations on splicing (Supp. Table S1). The Human Splicing Finder (HSF) combines results from several splice site and splice regulatory site prediction tools to estimate the impact of variations on splicing motifs [Desmet et al., 2009]. HSF and a commercial program ASSEDA [Nalla and Rogan, 2005] can predict the splice-site defect of exonic as well as intronic variations. MutPredSplice [Mort et al., 2014] and SKIPPY [Woolfe et al., 2010] predict splicing defects for exonic variations only.

Protein disorder predictors

Although proteins are often considered as static molecules, they undergo all kinds of fluctuations at different time scales. Some proteins or parts of proteins do not have an ordered structure at all. These intrinsically disordered regions and proteins are involved in several activities [Dunker et al., 2002]. The disordered regions are difficult to crystallize because they are highly flexible. Although numerous methods are available for predicting protein disorder, tools to predict the impact of variations on protein disorder were released only recently. Disease-causing amino acid substitutions are more often caused by disordered-to-ordered changes compared with the neutral amino acid substitutions [Vacic and Iakoucheva, 2012]. PON-Diso is a ML-based method for prediction of variation effects on protein disorder [Ali et al., 2014]. The method uses features representing evolutionary conservation and biochemical properties of amino acids.

Protein aggregation predictors

Protein aggregation is characterized by an increased level of β-sheet structure. Amyloid fibrils and amorphous aggregates are formed by protein aggregation. These fibrils and aggregates are caused by irreversible structural alterations and are associated with neurodegenerative diseases such as Alzheimer, Huntington, and Parkinson disease. Normal proteins can become toxic upon fibrillation [Bucciantini et al., 2004] and other proteins not related to amyloid diseases can aggregate under destabilizing conditions [Chiti et al., 1999]. Numerous tools predict amyloidogenic regions in protein sequences [Tartaglia et al., 2008; Garbuzynskiy et al., 2010; Maurer-Stroh et al., 2010; Emily et al., 2013]. Amino acid substitutions can increase the tendency of amyloidogenic proteins to aggregate. Methods for predicting the effect of amino acid substitutions on amyloidogenic proteins include AGGRESCAN [Conchillo-Sole et al., 2007], PASTA2.0 [Walsh et al., 2014], TANGO [Fernandez-Escamilla et al., 2004], and Aggrescan3D [Zambrano et al., 2015] (Supp. Table S1). Aggrescan3D requires three-dimensional protein structure to predict the effect of amino acid substitutions to aggregation, whereas the others utilize peptide sequences.

Protein solubility predictors

Protein solubility has been of great interest because of its relevance to protein (over)expression and for protein structural studies. Both nuclear magnetic resonance and X-ray crystallography require the protein of interest to be soluble in a high concentration. A low protein solubility has also been associated with diseases. Methods for predicting the effects of variations on protein solubility include OptSolMut [Tian et al., 2010], CamSol [Sormanni et al., 2015], and PON-Sol [Yang et al., 2016] (Supp. Table S1). OptSolMut and CamSol require a known 3-D protein structure, whereas PON-Sol is based on sequence context, evolutionary conservation, properties of amino acids, and other sequence-based features. Solubility and aggregation are related physical phenomena. Aggregation is an irreversible process due to structural and other changes, whereas precipitated proteins may be dissolved by dilution [Arakawa and Timasheff, 1985].

Additional mechanisms that amino acid substitutions can affect and for which prediction methods are available include, electrostatics, residue-residue contacts, side chain rotamers, cavity formation, and sterical clashes [Thusberg and Vihinen, 2009; Vihinen, 2015c]. When a residue is replaced, its chemical and physical properties may be altered causing structural alterations, especially when the original residue is smaller than the substituting one.

Method Performance Assessment

When choosing a prediction method, one should look at performance scores, preferably from independent benchmark studies. Three approaches can be used for testing method performance [Vihinen, 2012]. The first approach is challenge-based testing. Critical Assessment of Genome Interpretation (CAGI, http://genomeinterpretation.org/) is a challenge for method developers in the field of phenotypic impacts of genomic variation. Challenges do not aim for a systematic analysis of predictions, instead they assess what is currently doable, providing proof of concept, charting where to direct future efforts, and identifying new areas where predictive approaches would be needed.

The second test strategy is typically used by method developers. Most often the performance tests are not comprehensive, and the results are incomparable with those obtained from other methods due to using developer collected test sets. The third approach, systematic analysis, uses benchmark data and several evaluation measures to explain method performance.

Numerous measures should be used to describe predictor performance, since a single measure cannot capture all aspects of the performance. Typically, prediction methods are used as classifiers to define whether a case has the investigated feature or not. Results of this kind of binary predictions are presented in a 2×2 confusion or contingency table or matrix. Based on the four descriptors in the contingency table, several measures can be calculated (Fig. 3). Sensitivity, also called true positive rate (TPR) or recall, and specificity (true negative rate, TNR) show the ratio of the correctly predicted pathogenic and neutral cases. Positive predictive value (PPV/precision) and negative predictive value (NPV) are the conditional probabilities for pathogenic or neutral variants to be predicted as pathogenic or neutral, respectively. All these measures are calculated by using only half of the information in the contingency table. Accuracy and the Matthews correlation coefficient (MCC) utilize the whole matrix and are more balanced, representative, and comprehensive. To obtain a full picture of the predictor performance, it is important to evaluate all these six measures together [Vihinen, 2012].

Unless sufficient and similar numbers of positive and negative cases are used, the values of NPV and PPV may be biased, even meaningless. Accuracy and the MCC are affected by class imbalance only in extreme cases. The usual requirement is that the numbers of positive and negative cases be equal. To overcome the class imbalance problem, different approaches can be taken [Vihinen, 2012]. It can be done by pruning the size of the bigger class to that of the smaller one. It is also possible to normalize values in the contingency table. Normalizing makes sense only if dataset is representative.

Receiver operating characteristic (ROC) analysis is a visualization of prediction performance. It indicates the tradeoffs between sensitivity and specificity. The ROC curve can be drawn when the predictor provides a score for the classification. The faster the curve rises and the higher it reaches in the beginning the better the method is (Fig. 4). The area under the ROC curve (AUC) has been used as a measure of goodness for predictions as it approximates the probability of ranking a randomly chosen positive instance higher than a randomly chosen negative one. A value of 0.5 indicates a random classification, whereas 1 would indicate a perfect binary classifier. The ROC curve shows the method's ranking potential, which is related to overall performance.

If there are more than two predicted classes, the measures discussed above cannot be applied [Vihinen, 2012]. The data can still be presented in an N x N contingency table. Many performance scores used for binary classification can be generalized to multiclass case. The data could be divided into several partitions of two categories. It is possible to calculate row- and column-wise ratios. The MCC is a special performance score for binary data of linear correlation coefficient, which can be used for several classes in its general format. Generalized squared correlation (GC²) can be used for more than two classes [Baldi et al., 2000].

Problems and Remedies

Problems in the application of computational methods to variation analysis can appear at several stages and affect the interpretation of the results [Rogan and Zou, 2013; Vihinen, 2013, 2014a]. Before using any method, one should understand how it is used and for what purpose. As variation interpretation has become an ever more important topic, many methods have been developed. There has sometimes been a “me too attitude,” with people applying methods that have been developed originally in some other domain. These may cause problems, especially when the application domain and the datasets are not well known to the developers. Some quite poor datasets have been used for training many predictors. Methods developed in one application area can be useful in another domain when properly trained.

Another problematic area has been overfitting of data, see, for example, Grimm et al. [2015]. If the training data and features are not representative of the full spectrum of true cases, the generalization ability of an ML algorithm is poor and the method is overfitted. Overfitting may also occur due to other reasons that were described in an earlier section of ML methods. The same data should never be used for both training and testing [Vihinen, 2012] since it introduces circularity [Grimm et al., 2015]. Even usage of highly similar cases, such as variants in the same protein or protein family, may bias the method and provide good test performance but poor performance when applied to unseen real-life cases [Capriotti and Altman, 2011a; Bendl et al., 2014; Niroula et al., 2015].

Many variation effects are rare or very rare; therefore, it may be difficult to obtain sufficient numbers of cases for method training. It has been shown that the best predictor performance is obtained when using a balanced training dataset, that is, data that have equal numbers of positive and negative cases [Wei and Dunbrack, 2013]. If this is not done, the predictor will be biased toward the bigger class. For example, if a predictor is trained on a dataset containing more positive cases than the negative cases, the predictor will have a high TPR but a low TNR. The training dataset can be balanced by downsampling the bigger class, oversampling the smaller class or by assigning class weights [Vihinen, 2012]. It is also recommended to use a balanced test dataset for performance evaluation. However, the dataset imbalance does not have as high impact in testing as in training. Balanced performance scores such as balanced accuracy should be used in such cases.

It is not uncommon in the literature that authors present overblown statements about the quality of their methods [Vihinen, 2014a]. Therefore, it is important to look for independent quality assessments. Another problem is selective presentation of quality measures, showing only those that have a good performance for the tool. Many methods are not described in sufficient detail so that the readers could get a full picture about their quality. Many journals limit method details in articles such that the supplementary materials may be much longer than the actual article. In principle this is not problematic; however, quite often the supplements are not as carefully produced as the main text.

Human Mutation has published requirements both for prediction method developers and users [Vihinen, 2013] (see Boxes 1 and 2). For method users, these include the description of the choice and details of the method employed, its version, parameters and program options, and so on. In addition, program-generated statistical values should be provided. Special attention needs to be paid to the interpretation of the obtained results. Without all these details, readers cannot fully understand and evaluate the given results.

Choice of Tools

Whenever assessing variant pathogenicity, start by looking at known variants in databases including LSDBs, ClinVar and HGMD (especially if you have the commercial license), and others (Table 1). Many variants have heterogeneous effects; thus, these databases may not be fully conclusive. When using prediction methods, start with generic tolerance predictors or gene/protein/disease-specific predictors, if available (Supp. Table S1). In the next phase, mechanism/effect-specific tools can be applied to investigate the actual reason for the phenotype. Remember that no method is better than the input data; therefore, be aware of your variation data quality.

Choosing suitable methods can be a difficult task; however, it is important and will pay back. Reliable tools provide the best possible starting point for variation interpretation both for clinical and research purposes. Many tools are constantly being developed; therefore, it is important to follow the updates. Methods that were good a year or two ago may not have cutting-edge performance anymore. The community should not accept any black-box approaches; unfortunately, this is common with commercial solutions [Vihinen, 2015b]. Many companies do not provide sufficient details to allow understanding of how their methods work and what their true performance is. For a checklist of items to consider when choosing methods, see Box 3.

The Way Ahead

Progress in variation interpretation has been fast and is not likely to slow down, especially since the need is growing due to the increased application of sequencing technologies in research and clinics. Gene- and protein-specific tools will likely become more common and they can be trained when more data become available. There will still be a need for accurate tolerance predictors. Benchmark datasets need to be improved and expanded, developed for new application areas, and used for predictor training and testing. Only experimentally validated cases should be used for training. To continue to sell their products, companies need to become more open and unveil details about how their methods have been developed and how they perform on benchmark data.

Predictors should provide information that can be easily used for diagnostic decision making. Integration of various data items from several sources will be one of the themes in the future. This could create additional requirements for method developers, as all the data items may not always be available. Approaches to handle incomplete data have to be devised. Standardization of variation data and annotation has to be improved. The use of HPO and other ontologies for pathogenicity will allow easy utilization of phenotype data and VariO annotations will facilitate generation of accurate datasets for training and test purposes. Other standards are needed and they have to be applied strictly. An example of failed use of a standard data format is the VCF format, of which there are probably dozens of different versions that cause problems when transferring data between programs.

Genome sequencing is increasingly provided as a direct-to-consumer service. There are several problems related to data interpretation and especially in the communication of the significance of the findings. This should be done by professional counsellors, a service that these companies do not provide.

Protein and likely also RNA structural information will be increasingly used for the interpretation of variant effects. There are already tools available, but the throughput speed is usually not suitable for large-scale studies. Structural insight will allow for new ways of understanding variation effects as well as for designing therapies to treat pathologies. For example, many variants that have stable structures affect molecular interactions [Sahni et al., 2015].

Bioinformatic tools are increasingly compiled into pipelines and workflows, such as Galaxy [Goecks et al., 2010], Taverna [Wolstencroft et al., 2013], and Anduril [Ovaska et al., 2010]. They have numerous benefits, especially in routine use. Users also have to be aware of situations when the pipelines are inappropriate to use. Cloud computing has been a buzz word in recent times. For most users of variation interpretation tools, it does not matter where the data are analyzed. But for clinical applications, security is a prime question. Due to potential security and privacy issues with distributed computing, interpretation via cloud services may need to be approached according to local and national laws. Indeed, improvements in online security are being made at constant pace.

Quality of interpretation in prediction algorithms will likely be the issue of greatest importance in the future. In fact, it applies to the whole process of genomic medicine. The smallest possible error rate must be achieved in sequencing, variant calling, annotation, and interpretation. Prediction methods have to be highly accurate, and at the same time very fast, to cope with exome and complete genome sequence data for numerous individuals. Interpretation has become and likely will remain the bottleneck for the use of variation data in clinical practice. Intelligent solutions are already available and will be further improved in the future. Computer-assisted variation interpretation will be part of the toolbox for everyone working on sequencing data generation and interpretation.

Disclosure statement: There is no conflict of interest to declare.