CardioVAI: An automatic implementation of ACMG-AMP variant interpretation guidelines in the diagnosis of cardiovascular diseases

2.1 Criteria implementation and scoring system

Seventeen of 28 criteria proposed by the original ACMG–AMP guidelines have been implemented in CardioVAI (see Supporting Information Table S1). Despite we did not include the automatic evaluation of the other criteria especially because referring to patient specific information (such as co-segregation of the variant in family members), its manual inclusion is supported (see Web Interface section). Notably, we defined and added a new supporting benign criterion (BP8, see Supporting Information Table S1). Moreover, we have incorporated recent adaptation of ACMG–AMP guidelines for MYH7-related variants (Kelly et al., 2018) proposed by the ClinGen Cardiomyopathy Expert Panel (CMP-EP). As a consequence, MYH7 variants will be classified according to CMP-EP recommendations.

In the classification process, each variant is associated to a list of CVDs, known to be related to the corresponding gene through MedGen database. Each variant-disease or variant-phenotype association undergoes two interpretation steps. First, the tool systematically checks the implemented criteria, then final class is assigned depending on the number of verified criteria and corresponding level of evidence according to ACMG–AMP guidelines.

Our implementation of population frequency criteria, i.e., PM2, BA1, and BS1, relies on three different databases: ExAC, 1000 Genome Project Phase 3 (1TGP), and ESP that are based on different set of sequenced genomes. For pathogenic criterion, if the variant has an allele frequency higher than expected in at least a database, PM2 is not triggered. Conversely, benign criteria are triggered in case variant allele frequency is higher than a threshold in at least a database. Selected frequency thresholds are equal for ExAC and 1000 Genome Project, while ESP threshold is higher since its patients cohort may include several cases with CVDs phenotypes (Auer et al., 2016). Allele frequency threshold has been set to 5% for BA1 for every population database, 0% in case of a dominant disease, and 0.01% in case of a recessive disease for PM2 (0 and 1% for ESP database, respectively). BS1 allele frequency threshold varies according to Orphanet database and in case of a recessive disease for which the carrier frequency is not known, Hardy Weinberg equilibrium is used for its estimation and to set the corresponding frequency threshold. Table 1 reports two CVD examples with corresponding thresholds used for BS1. For MYH7 variants, frequency thresholds are those reported by the CMP-EP (see Supporting Information Table S1).

In silico tools selected for implementing PP3 and BP4 are PaPI, DANN, and dbscSNV. The first one combines pseudo amino acid composition (PseAAC), Polyphen2, and Sift to predict and score the functional effect of single nucleotide and short insertions/deletions coding variants (Limongelli et al., 2015). DANN provides a score for variants that occur in noncoding regions as well (Quang et al., 2015), while dbscSNV is useful for the prediction of both coding and noncoding variants occurring nearby splicing sites (Jian et al., 2014).

As a reference for bona fide pathogenic and benign variants, necessary for PS1, PS3, PM5, PP5, BS3, BP6, and BP8, we based on ClinVar repository (version 2017-03). We excluded variants with conflicting interpretation, i.e., variants interpreted both as “Pathogenic/Likely pathogenic” and “Benign/Likely benign” from different submitters. The resulting set includes 21 117 submitted variants in genes associated with CVDs, 4160 were interpreted as “Pathogenic” or “Likely pathogenic” without conflicts, 4947 benign or likely benign variants, and about 670 variants with conflicting interpretation among submitters. Table 2 shows some examples of variants with conflicting interpretation in ClinVar that were not considered for the triggering of related criteria. However, according to CMP-EP, PP5 and BP6 are removed for variants in MYH7 gene.

We then collected more than 200 functional domain regions implicated in CVD disorders from literature. These have been used for PM1 criterion implementation. For instance, we included a hotspot region in DSG2 gene (Kapplinger et al., 2011). Table 3 reports a list of included functional domains.

For PVS1 implementation, about 41 genes associated with CVDs LOF/GOF are well-known mechanism of disease (See Supporting Information Table S2). For the remaining, we collected from ExAC probabilities of being intolerant to LOF mutations in heterozygous (pLI) or in homozygous (pRec) (Lek et al., 2016). For MYH7 variants, this criterion level of evidence has been downgraded to a moderate level of evidence, as stated by the CMP-EP.

Finally, we translate ACMG–AMP levels of evidence (very strong, stand alone, strong, moderate, and supporting) of each class (benign or pathogenic) into scores (Table 4). Final pathogenicity score for each variant-phenotype association is the sum of scores of the triggered criteria. We anticipate that pathogenicity score can be used to provide a rank of variant phenotypes, particularly useful to evaluate VUS and prioritize possible overlapping phenotypes associated to a certain variant (e.g., Long QT vs. Brugada Syndrome for SCN5A gene variants).

Detailed information about criteria implementation could be found in Supporting Information Table S1.

2.2 Benchmark datasets

In order to test CardioVAI performance, we collected two benchmark datasets of previously interpreted variants from online resources (CardioDB and CLINVITAE).

CardioDB (Atlas of Cardiac Genetic Variation https://cardiodb.org/ACGV/) contains a list of pathogenic variants related to hypertrophic cardiomyopathy, dilated cardiomyopathy, and arrhythmogenic right ventricular cardiomyopathy. Pathogenicity assessment is performed by the Oxford Molecular Genetics Laboratory (OMGL) and the Laboratory of Molecular Medicine (LMM).

The second dataset is CLINVITAE (https://clinvitae.invitae.com/), an online database that collects interpretation of variants related to a broad set of diseases from different sources, included ClinVar, ARUP Mutation Database, Carver Mutation Database, and Emory Genetics Laboratory Variant Classification Catalog.

In both datasets, we excluded variants with conflicting interpretation among sources. Variants whose interpretation was not clearly indicated in one of the five ACMG–AMP classes were excluded. For example, in CardioDB dataset we filtered out variants interpreted as “VUS favor pathogenic”, while in CLINVITAE variants with status “Suspected benign” or “Unclassified” were kept out.

From CardioDB, we extracted 93 variants in eight CVD genes whose pathogenic interpretation has been assessed both by OMGL and LMM laboratories.

In CLINVITAE dataset, we kept only 58 CVD-related genes (see Supporting Information Table S2). The resulting dataset has 885 variants, 200 interpreted as pathogenic or likely pathogenic, and 685 interpreted as benign or likely benign (see Table 5).

3.1 Benchmark analysis

Each collected variant undergoes CardioVAI classification process. A variant may be interpreted as Pathogenic for a phenotype and VUS for another. Therefore, we labeled as “Pathogenic” or “Likely pathogenic” (P/LP) variants with at least one pathogenic or likely pathogenic phenotype association. The same logic is applied to “Benign” or “Likely benign” variants (B/LB). If a variant has pathogenic and benign association for two different phenotypes, it is considered to belong both to P/LP and B/LB classes. P/LP and B/LB variants were considered as positive and negative classes, respectively, while predicted VUS have been considered as error type I/II. Herby, we will use sensitivity and specificity terms according to these definitions.

Table 6 reports pathogenic and benignity concordances of CardioVAI classification on CardioDB and CLINVITAE benchmark datasets. CardioVAI reached an average sensitivity of 74.8% and a specificity of 97.08%. Because CardioDB exclusively reports pathogenic variants, specificity has been calculated on CLINVITAE dataset only.

Figure 1a and b show Venn diagrams of benign and pathogenic CLINVITAE variants classified by CardioVAI. Notably, all those misclassified variants in CLINVITAE have been predicted as VUS, therefore there were no truly pathogenic variants classified as B/LB or truly benign variants classified as P/LP. Instead, only one pathogenic variant (MYH7:p.Arg442Cys) in CardioDB benchmark was misclassified as LB (see Venn Diagram in Figure 1c). This variant occurs in MYH7 gene, therefore it has been classified according to CMP-EP adapted criteria. Since variant is observed in 1TGP database with an allele frequency of 0.3%, BS1 (stand-alone criterion for “Likely benign” classification in CMP-EP adapted rules) has been triggered.

In order to understand which implemented criteria contributed more to the final classification, we built a differential criteria heatmap on CLINVITAE data. Since 17 MYH7 variants were classified according to CMP-EP adapted criteria, we excluded such variants from the criteria heatmap analysis. We divided variants into four groups, according to benign/pathogenic concordance and discordance (as shown in Figure 1a and b). For each group, scores representing the mean of criterion activation in CardioVAI for that group are computed. Results are shown in Figure 1d. Both for benign and for pathogenic concordance groups, the more frequent triggered criteria are those relying on bona fide variants collected from ClinVar (PP5, BP6, and PS1) and those based on in silico tools combination (PP3 and BP4). This result confirms that our selected methods for damaging prediction well support pathogenicity evidence. Other frequent criteria rely on population frequency (BS1 and PM2), while the activation of PM1 criterion in pathogenic concordance groups confirms the importance of the functional domains collected from literature in pathogenic detection. In discordant groups, we can observe that despite the misclassification, active criteria's type (benign/pathogenic) is concordant with the true classification in CLINVITAE. Only PP3 criteria is often triggered in benign variants as well, suggesting that in silico tools are more biased toward a protein-damaging effect of variants.

To assess whether the introduction of a pathogenicity score could help in VUS prioritization, we evaluated the distribution of pathogenicity scores of benchmark classified variants that CardioVAI predicted as VUS. The total number of predicted VUS variants is 84 (54 from CLINVITAE and 30 from CardioDB, about 20 benign and 64 pathogenic variants). The distribution of pathogenicity score for these VUS variants is shown in Figure 2. Scores distribution multimodality is confirmed by the dip test of unimodality (Hartigan & Hartigan, 1985) (dip test index D = 0.10714, P-value < 2.2 × 10^–16). Moreover, since P-value is less than 0.05, we can confirm that scores distribution is at least a bimodal distribution (Freeman & Dale, 2013), thus reflecting the actual classification in benchmark datasets. Finally, we have carried on a ROC analysis (see Supporting Information Figure S1) of the pathogenicity score on the whole CLINVITAE benchmark dataset, obtaining an AUC of about 91.43%.

3.2 Comparative analysis

We compared our approach to InterVar and CardioClassifier, two similar ACMG-based automatic tools while excluding from the comparative analysis tools like ClinGen Pathogenicity Calculator since the user is needed to manually select criteria for each variant.

Since InterVar and CardioClassifier do not incorporate CMP-EP interpretation framework, we excluded from benchmark dataset 42 MYH7 variants from CardioDB and 17 MYH7 variants (five pathogenic and 12 benign) from CLINVITAE. In fact, CardioVAI interprets MYH7 variants according to CMP-EP specific recommendations, which substantially refined ACMG–AMP guidelines. In order to avoid classification comparison made with different interpretation systems (the ACMG–AMP guidelines and the CMP-EP recommendations), we excluded MYH7 variants from comparative analysis.

Both InterVar (version 0.1.7) and CardioClassifier (accessed in November 2017) were run with default settings.

InterVar is a freely available semi-automatic pipeline (Li & Wang, 2017). As for CardioVAI, ClinVar is the reference database for well-established pathogenic/benign variants. The number of common implemented criteria is 17. In addition, InterVar implements BS2 benign criterion. We ran InterVar on 51 pathogenic variants from CardioDB. Compared to CardioVAI that was able to correctly classify 46/51 pathogenic variants, InterVar correctly detected only 23/51 pathogenic variants. Figure 3a shows CardioVAI and InterVar results on CardioDB benchmark dataset upset diagram with intersected variant classifications. Figure 3b shows differences in criteria activation between CardioVAI and InterVar for those variants that were correctly interpreted by CardioVAI but not by InterVar.

CardioClassifier (https://www.cardioclassifier.org/) is a web tool developed by the Cardiovascular Genetics and Genomics team of the Imperial College London. It is able to automatic classify variants related to CVDs implementing 17 ACMG-AMP criteria, but is limited to 40 genes only. Therefore, we compared CardioVAI to CardioClassifier on a CLINVITAE variants subset, corresponding to those variants in these supported CVDs-related genes. CardioDB variants were not included in the comparison analysis in this case since they have been all used as reference for bona-fide pathogenic variants by CardioClassifier, thus avoiding a biased benchmark analysis. We collected 462 CLINVITAE variants (166 pathogenic and 296 benign) in 13 genes that could be classified by CardioClassifier. Concordance with CLINVITAE benchmark dataset for CardioClassifier, CardioVAI, and InterVar is shown in Figure 4. Despite all three classifiers reported high benignity concordances (greater than 90%), CardioVAI showed the highest sensitivity (84.9%) and specificity (96.96%). Moreover, compared to CardioClassifier and InterVar, CardioVAI was able to reduce the number of VUS of about 65.3% (from 98 to 34 VUS) and 70.9% (from 179 to 52 VUS), respectively. See Figure 3 for upset diagrams showing the classified variants intersections between the methods.

In order to better understand which criteria contributed to different performances between classifiers, we analyzed InterVar and CardioVAI classification on CLINVITAE dataset. InterVar output is a tab-delimited file, which contains the list of triggered criteria for each variant and it could be easily processed. Because CardioClassifier does not allow results to download, it was not possible to perform such analysis with this tool.

Figure 3c and d show CLINVITAE benign and pathogenic variants classified by InterVar and CardioVAI respectively. Variant classification mismatches between the two methods (those correctly classified by CardioVAI and not by InterVar and viceversa) were grouped into a differential criteria heatmap (Figure 3e) to highlight the differences between the two tools in terms of ACMG–AMP activated criteria.

Remarkable differences have been observed in criteria based on bona fide pathogenic/benign variants knowledge base (i.e., ClinVar) criteria, i.e., BP6, PP5, and PS1. Differences could be explained by the use of different ClinVar versions (201703 vs. 20160302) and the way of database processing (e.g., normalization and use of variants with conflicting interpretations). Notably, CardioVAI is very stringent with ClinVar interpretations of pathogenicity, requiring the presence in ClinVar of the phenotype for which the assessment has been done. BS2 criterion, not implemented by CardioVAI, contributes to concordant benign classification, while our introduced BP8 criterion did not affect classification in this particular dataset. The analysis highlights the well-known flexibility in ACMG–AMP criteria implementation (Hoskinson et al., 2017): discordant activation in BP1 and PP2 criteria is due to different implementation strategy by CardioVAI and InterVar in the definition of “missense gene” and “truncating gene” (genes for which primarily missense/truncating variants are known to cause disease). PP3 criterion (in-silico prediction tools) contributes to the pathogenicity assessment gap between the two tools as well, potentially justified by the different tools used (PaPI and DANN for CardioVAI and MetaSVM and GERP++ for InterVar). However, the PP3 criterion results overrepresented for benign variants too, thus confirming that in silico tools have a bias toward protein-intolerance prediction (Ghosh, Oak, & Plon, 2017).

3.3 Validation on MYH7 variants assessed by ClinGen Expert Panel

We further validated CardioVAI by comparing its ACMG–AMP-based classification on 60 MYH7-related variants assessed by ClinGen's Inherited Cardiomyopathy Expert Panel (CMP-EP) (Kelly et al., 2018). In this work, ACMG–AMP guidelines have been applied to MYH7 gene, providing a set of adjusted criteria and final classification rules. Refinements includes specific allele frequencies thresholds for BA1 and BS1 and changes in criteria levels of evidence. For instance, PVS1 criterion has been declassed from “very strong” to “moderate”. Final rules for five-tier classification were also revised. For instance, BS1 criterion has been considered a stand-alone criterion to classify a likely benign variant (see Supporting Information Table S1). Moreover, family co-segregation and case-control studies have been additionally used in CMP-EP.

CardioVAI implements ACMG–AMP guidelines revised for CMP-EP specification on MYH7 variants and we compared the results on the 60 expert-panel validated variants. Since these variants were interpreted by the CMP-EP using co-segregation data as well, we activated corresponding criteria to the final classification. We achieve a concordance of 93.4%: only four variants out of 60 were incorrectly classified by CardioVAI mainly due to different population databases used (1TGP, ExAC, and ESP for CardioVAI and ExAC only for CMP-EP).

The p.Glu1902Gln variant is classified as LB by CardioVAI, since it is reported in 1TGP at allele frequency (0.039%) greater than the proposed threshold (0.02%) and therefore CardioVAI triggered BS1 criterion which is stand alone for LB interpretation. CMP-EP reports the same variant as VUS, since it is reported at a lower allele frequency in ExAC only (0.0074%). Interestingly, another variant (p.Arg787His) has been classified as LB by CardioVAI since it is reported in ExAC with frequency 0.022% (thus activating the BS1), however the same variant has been interpreted as VUS by the CMP-EP and BS1 was not triggered in this case (Kelly et al., 2018). The third case is an LP variant that CardioVAI classifies as VUS (p.Arg1045Cys). The missing criteria for classification is PM2, since the variant is present in 1TGP at higher frequency (0.01997%) than the threshold set by CMP-EP for this criterium (0.004%). Finally, p.Arg1420Trp is reported as LP by the CMP-EP panel and triggers two pathogenic moderate criteria and one pathogenic supporting criteria (PM2, PS4 declassed to moderate, and PP3). However, according to CMP-EP final rules, the variant should not reach sufficient evidence for pathogenicity as reported by CardioVAI. See Supporting Information File S2 for further information.

3.4 CardioVAI on all missense variants in CVDs-related genes

We collected about 600 000 variants in 72 genes associated with CVDs from dbNSFP (Liu, Wu, Li, & Boerwinkle, 2016), a database of all possible non-synonymous single nucleotide variants in human genome. Genes were selected according to three different resources: the TruSight Illumina Cardiomyopathy Panel, CardioDB, and our in-house knowledge base. The complete gene list and the corresponding sources are reported in Supporting Information Table S3. Among 72 genes, CardioVAI identified about 11.6% of pathogenic/likely pathogenic variants for at least a CVDs disorder. About 25.2% are interpreted as benign or likely benign (Figure 5a).

Among 72 genes, the most significantly classified are shown in Figure 5b.

Highest percentages of pathogenic/likely pathogenic variants were found in KCNH2, SCN5A, LMNA associated with Long QT syndrome, Brugada syndrome, and Familial dilated cardiomyopathy. respectively. Conversely, very few pathogenic variants were reported in genes such as LAMA4 and MYPN that are associated to dilated cardiomyopathy both in CardioDB and MedGen. A possible explanation of the low rate of pathogenic-predicted variations could be the tolerance that these two genes show to missense variants (that represents the majority of CardioVAI interpreted variants). In fact, both genes have an ExAC missense zscore lower than 0 (–0.35 for MYPN and –0.67 for LAMA4).

We studied the correlation between the percentage of each gene pathogenic variants with the number of associated phenotypes (see Supporting Information Figure S2). Pearson correlation coefficient is about 0.76, denoting that genes with higher percentage of pathogenic variations are more associated with a higher number of phenotypes, thus reflecting their important role in different biological mechanisms related to CVDs.

3.5 Web interface

We developed a web resource (http://cardiovai.engenome.com) to query CardioVAI precomputed classification of all non-synonymous variants in 72 CVDs-related genes (see Figure 6). Users can search a variant by genomic coordinate or HGVS nomenclature (coding or protein) and customize ACMG–AMP rules for final classification. Each row of the resulting table corresponds to the interpretation of a certain variant phenotype. Rows are ordered by pathogenicity score and additional information about variant annotation (e.g., gene, ClinVar, population databases, and in-silico tools predictions) and classification within its triggered criteria are provided. ClinVar cross-references are shown in case a variant matches with a ClinVar submissions, both at the nucleotide and amino acid level. For each variant-phenotype, all 28 ACMG-AMP criteria are listed and users can adjust the final classification by manually trigger or deactivate specific criteria and their level of evidence. User is allowed to download a tab-delimited file that contains CardioVAI interpretation of the variant, but also the entire precomputed classification of all non-synonymous variants.