Increasing phenotypic annotation improves the diagnostic rate of exome sequencing in a rare neuromuscular disorder

2.1 Editorial policies and ethical considerations

The RD-Connect GPAP has received ethics approval from the Parc de Salut MAR Clinical Research Ethics Committee on 27 October 2015 under Ref. No. 2015/6456/I. All data used in this study had been originally submitted to RD-Connect under the ownership of the study principal investigator (PI), H. L., and was accessed by authorized researchers in compliance with the RD-Connect Code of Conduct. As with all data in the RD-Connect GPAP, no individuals in this study were personally identifiable and all had been consented by the submitting clinician to analysis of their data for research purposes in accordance with local ethics regulations.

2.2 Patient cohort

Since our aim was to explore the effect of varying phenotypic annotation on the solve rate in a real patient cohort, we needed to select a set of patients whose cause of disease was already known to establish our ground truth, and to minimize nonphenotypic variables to focus on the effect of phenotype. From the cases available within the study PI's cohort in the RD-Connect GPAP, we selected a restricted cohort of 29 individuals with a confirmed genetic diagnosis (“solved cases”), all of which had a similar phenotypic profile and homozygous causative variant. All individuals had originally been clinically suspected of a congenital myasthenic syndrome (CMS; Thompson et al., 2018) based on their phenotypic presentation to the submitting clinician and had been submitted within the RD-Connect PhenoTips instance as CMS cases. All were subsequently diagnosed with a homozygous recessive cause of disease that was confirmed by further analysis by the submitting center. Our review of the cases confirmed the compatibility of the phenotypic description with CMS. The aim of restricting the initial analysis cohort in this way was to minimize the effect of the genetic variation on the analysis to focus on the effect of phenotype on the analysis results. The 29 cases covered 18 different genes in total (see Table 1). Twenty two of the twenty nine cases (11 genes) had causative variants in genes that have been previously associated with CMS according to the Online Mendelian Inheritance in Man (OMIM) database (Amberger, Bocchini, Scott, & Hamosh, 2019); 5 cases (5 genes) had variants in genes that were associated with other human disease but not CMS; and 2 cases (2 genes) had no human disease associated according to OMIM but have been assigned as causative by the submitting clinician on the basis of segregation and downstream analysis. The exomes analyzed came from several different projects and were not all from the same original sequencing provider but had all been reprocessed from the raw data (FASTQ or untrimmed BAM) using the standard RD-Connect analysis pipeline (Laurie et al., 2016) to standardize variant calling and annotation in line with standard RD-Connect practice.

Knowing the confirmed diagnosis of these cases allowed us to assess the success of different analyses in terms of their ability to prioritize the correct causative variant. A standard workflow recreating the recommended analysis approach for a new case in RD-Connect was applied to all cases. First, the total number of variants in each exome before applying filters was assessed. Then, to reduce the number of variants to evaluate in the phenotypic part of the analysis, each exome was filtered using a standard set of genomic filters available within RD-Connect, including variant class and population frequency (Table S1). At each stage it was confirmed that the confirmed pathogenic variant was still present in the resulting data set and had not been filtered out. On achieving a filtered data set of rare, plausibly pathogenic variants based on variant data alone, we applied several methods of prioritizing by phenotype, as described in Experiments 1 and 2 below.

2.3 Standardized HPO set

All individuals selected for the analysis had varying numbers of HPO terms entered by the submitting clinician at the time of submission (ranging from 1 to 18 terms, all specific to that individual), and these were used as part of our analyses as described below. In addition to these submitter-entered HPO terms, we also generated a standardized or simulated set of seven HPO terms using phenotypic features that are common to many of the CMS types and frequently used to describe CMS patients. This standardized set was designed to be applied across all cases to enable us to assess the effect of varying the phenotypic annotation. In this standardized set we endeavored to cover a range of different branches of the HPO hierarchy, including neurological, muscular, respiratory, joint, and eye phenotypes. It was intentionally generated by expert knowledge without reference to the existing HPO gene:phenotype annotations in order not to be biased towards known annotations. We applied the standardized list in Experiments 1b and 2 below. An overview of the experimental workflow is provided in Figure 1.

2.4 Hierarchy of terms and information content

Our initial standardized set of terms was selected to cover clinically appropriate CMS annotations at a level of specificity that reflects common use among CMS experts. Since the HPO represents these terms within a hierarchy, it is possible to explore the way that changing the level of specificity affects the performance of the prioritization tools by using terms that appear at higher levels in the hierarchy, We define “parent terms” as those appearing one level up in the hierarchy from our original terms, and “grandparent terms” as those appearing two levels up. The information content (IC) of each HPO term can be calculated as a function of the frequency with which that term or its subclasses appear as annotations for the diseases in the HPO disease:phenotype database (Groza et al., 2015). The IC thus reflects the clinical specificity of the term (broader terms appear more frequently when their subclasses are also included, while more specific terms are less frequent) and is an important component of the algorithms underlying the Exomiser tools. It is calculated as the negative log of p_t, where p_t is the frequency of occurrence of the term among all the annotations in the HPO disease:phenotype database.

Using disease and annotation numbers available in the online HPO browser (https://hpo.jax.org/app/browse/) we calculated the IC for each of our seven terms in the standardized set. To assess the performance of the algorithms with more general terms with lower information content, we then extracted the parent terms and grandparent terms and calculated the IC in the same way. The values are shown in Table 2.

2.5 Experiment 1a: Disease-gene list

A disease-gene list or “virtual panel” is commonly used in clinical diagnostics where exome sequencing has been performed. This restricts the output to variants in a predefined set of genes that have already been associated with human disease with a particular global phenotypic presentation. Several lists curated by disease domain experts are available within the RD-Connect system and can be applied as a filter on the data. Since our cohort all had a clinical starting diagnosis of CMS, which is a neuromuscular disease (NMD), we applied the 2016 version of the Neuromuscular Gene Table list (Kaplan & Hamroun, 2015), a comprehensive and carefully curated set of genes associated with NMD which has become the de facto standard for the neuromuscular field. To evaluate this filtering method, we defined our detection rate as the proportion of cases in which the known causative variant was among the variants returned. We also assessed the total number of variants returned.

2.6 Experiments 1b and 1c: Gene lists generated from HPO terms

As an alternative to a virtual panel for a disease domain, it is possible to generate a gene list by compiling all genes that have been annotated in the literature as associated with an individual HPO term or collection of terms. This makes use of curated gene:phenotype annotations compiled by the HPO developers and updated on a monthly basis. A PhenoTips-developed RESTful API (The PhenoTips developers, 2017) enables the compilation of a list of genes based on a given set of HPO terms. Our analysis used the annotation set from September, 2018 (The HPO developers, 2018). Here, we performed two analyses. First, we took the actual phenotypic terms entered by the submitting clinician and used the API within RD-Connect to generate a gene list from those terms. Since each case had a unique annotation set, this created a unique gene list for each patient. Second, we used the same RD-Connect function with our seven standardized HPO terms, which enabled us to apply a consistent set of terms and therefore a consistent gene list across the 29 cases. Again our detection rate for both cases was the proportion of cases in which the known causative variant was returned.

2.7 Experiment 2. Variant prioritization using the Exomiser

The Exomiser is an application designed to find and prioritize potential disease-causing variants in WES and WGS data (Smedley et al., 2015). As input it takes a VCF file and a set of phenotypes in HPO, and from this it annotates, filters and prioritizes likely causative variants. It creates a variant score from predicted pathogenicity and population frequency data and a phenotypic relevance score based on a comparison of how closely the given phenotype matches the known phenotype of disease genes. The two scores combined are then used to rank the candidate variants in terms of the likelihood that each may be causative. The phenotypic relevance score can be set to base its results on human data only (PhenIX algorithm), model organism data only (PHIVE), or human, model organism and interactome data combined (hiPHIVE). Mode of inheritance may be entered to include only those results compatible with the assumed inheritance pattern. By providing a score and a ranking for each variant with each phenotypic combination, the Exomiser results provide a more sophisticated method for evaluating the effect of phenotypic annotation beyond the inclusion/exclusion method provided by the gene list experimental setup.

In the RD-Connect GPAP, users have the possibility to submit candidate variants and associated phenotypes to Exomiser for analysis, but it can only be run with the HPO terms entered by the submitting clinician, and not with our standardized set of terms. For this test, therefore, we installed a separate Exomiser instance on a local server. We extracted the filtered candidate variants from the RD-Connect platform in VCF format and used them in Experiments 2a and 2b as follows.

2.8 Experiment 2a: Variant prioritization using the Exomiser with its human data comparison algorithm (PhenIX)

Using the VCFs containing the filtered variants together with the standardized HPO set described above, we set Exomiser to its human-only algorithm (PhenIX) and homozygous recessive inheritance pattern and sequentially ran all possible variations of cases and HPO terms, from one HPO term to seven terms in the standardized set in combination with all 29 cases (3,683 separate combinations). We then repeated the test with the “real” HPO set for each case as entered by the submitter (29 further experiments). We captured the output for the known causative variant, including the gene-variant score (genomic data alone; no consideration of phenotype), gene-pheno score (the match of the phenotype with the gene at gene level, excluding variant pathogenicity assessment), and gene-combined score (combines the variant and phenotype scores to give an overall assessment of likelihood that the variant is causative). We also captured the rank of the causative variant in each case.

To assess whether the level of specificity of our annotations affected the score and rank obtained using the Exomiser algorithm, we also repeated the analysis using the parent and grandparent terms, capturing the same scores and ranks for comparison against those obtained using our original annotations.

2.9 Experiment 2b: Variant prioritization using the Exomiser with its human, model organism and interactome comparison algorithm (hiPHIVE)

We repeated the same experimental setup as for 2a, the only difference being that Exomiser was set to run with the hiPHIVE algorithm. The same set of results was captured and compared against the data for the PhenIX algorithm to provide an assessment of whether and under what circumstances it is beneficial to include comparison against model organism and interactome data.

3.1 Overall results and performance of gene list filters

As expected, before filtering, all exomes started with large numbers of homozygous variants (mean: 59,236; range: 25,626–81,527). After filtering by variant class high/moderate/low (i.e., excluding modifiers) and setting a population frequency filter of 0.001 in ExAC (Lek et al., 2016) and filtering by homozygous inheritance pattern, the variant number reduced to a more manageable size (mean: 34 variants; range: 6–102) with the causative variant still present in all cases. In Experiment 1a, filtering with the 2016 version of the neuromuscular gene table as a virtual panel (416 genes) retained the confirmed causative gene in 23 of 29 cases, resulting in a detection rate of 79.3%. Of the six cases where the neuromuscular gene table did not identify the causative gene as a possibility, in three cases it provided no candidates, while in three more cases it provided other incorrect genes while excluding the correct one (false positives). The lists based on HPO terms retained the confirmed causative gene in 22/29 cases (standardized HPO list resulting in a list of 616 genes) and 23/29 cases (real HPO list for each case), detection rates of 75.9% and 79.3%, respectively, very similar to the neuromuscular gene table results. As expected, the gene list filters were unable to pick up the new CMS genes in the cohort and this strategy thus naturally fails in a discovery paradigm. Of the cases where the correct result was not returned, the standardized HPO list provided false positives in five of seven cases (standardized list) and two of six cases (real list). The overall false positive rate is 10.3% for the neuromuscular gene table list, 17.2% for the standardized HPO list, and 6.8% for the real HPO list. In 3/29 cases the causative gene was not returned by any of the three gene-list methods.

3.2 Exomiser

The Exomiser results are not directly comparable with the gene list results, since Exomiser assigns variants a priority score rather than including or excluding them based on a predefined list. When considering the variant alone, Exomiser provides a variant score based on population frequency and pathogenicity inferences. Using this score and the associated ranking compared with the other variants evaluated, we could assess how well Exomiser performed in the absence of phenotypic data, and thus what added value the phenotypic data provided. We see (Table 3) that using variant data alone, the confirmed causative variant ranked in the first position in 18 out of 29 cases (62%). With all seven HPO terms considered and using the (human data) PhenIX algorithm, this increased to 26 out of 29 cases (89.6%). This indicates that even though we are dealing with variants that have been prefiltered to be highly plausible, the addition of phenotypic data does improve the ranking of the correct variant. Our experimental design then enabled us to look more deeply into the data to analyze the numbers and combinations of terms and genes that performed well or poorly, and the difference between human and animal data.

3.3 Exomiser with known CMS genes: Using PhenIX (Experiment 2a)

Analyzing the Exomiser rankings and scores in further detail provided some findings that allowed us to further stratify the data. Exomiser results are provided to the user with a ranking according to a combined score ranging from 0 to 1 that combines both the variant and the phenotypic score, but the variant-only score and phenotype-only score are also presented. Our results with the PhenIX algorithm showed a global difference between known CMS genes and those not associated with CMS, as is to be expected from the gene:phenotype annotations on which Exomiser bases its human-data algorithms. We therefore analyzed the known genes separately and found that although the variant-only ranks were already generally high, reflecting a strong candidate from the genotypic perspective, the ranking of the known CMS genes did further improve even with a small number of phenotypic annotations: on adding a single HPO term, the mean rank improved from 1.6 (variant alone) to 1.09 (one HPO annotation). In addition, we found that even with these high ranks, adding further HPO terms progressively improved the rank until all cases had achieved a rank of 1.0 (Figure 3a), which occurred at 6 HPO terms.

However, evaluating not only the rank but also the score itself is also an important part of the analysis of the results: while the rank is a measure of the performance relative to the other variants being analyzed, the score is a measure of that variant's estimated pathogenicity and its match with the phenotype completely independently of the other variants present. Given that a “perfect” score would be 1.0 and that highly plausible variants in known disease genes frequently achieve combined scores as high as 0.997, it is evident that a low score of for example, 0.02 would raise questions about the likelihood of a variant's pathogenicity even if it were in first position in terms of ranking. Across all cases with known CMS genes, the score with a single HPO annotation ranged from 0.0695 to 0.923 (minimum and maximum mean values). It is evident from these results that it is perfectly possible for a variant to achieve a top score with even a single phenotypic annotation if this annotation happened to “hit the mark” and match the gene:phenotype annotations perfectly. It is also clear that with fewer annotations, the algorithm may be misled. We, therefore, also wanted to assess whether additional phenotypic annotation increased not only the rank but the likelihood of achieving a plausible score. Setting a value of 0.5 for the combined score as a reasonable threshold derived from user experience and combining that with the desirability of achieving a top rank, we showed that the probability of achieving both a score > 0.5 and a rank of 1 increased from 0.63 to 0.94 as the number of HPO annotations increased from one term to seven (Figure 3b). Extrapolating the trend line with higher HPO numbers suggests that the probability would reach unity at around nine HPO terms. Figure 3 further suggests that increasing the number of HPO terms to five or above increases the reliability of the analysis when it comes to known genes: not that it is not possible to achieve a high rank and score with fewer, but rather that the outcome is less likely to be incorrectly skewed by a phenotypic term that happens not to be associated with the gene in question. In addition to the progressive increase in the score of the known causative variant with increased annotation numbers, we observed a similar progressive decrease in the mean score of the second-ranked variant when analyzing the subset of cases where the correct variant was consistently ranked top. When going from one to seven annotations, the mean score of the top-ranked variant increased from 0.50 to 0.57, while that of the second-ranked variant decreased from 0.059 to 0.017 (Table S2 and Figure S1). This shows that additional annotation has the effect of increasing the confidence in the top result by increasing the distance between the top and second-ranked variants.

We then looked into the maximum and minimum scores obtained across all combinations to assess the variability of the combined score with different levels of phenotypic annotation. For each case we plotted the highest and lowest scores achieved with any number of HPO annotations from the standardized set, as well as the score achieved with the real submitter-entered HPO annotations, and the highest standardized score with a number of HPO terms equal to the number entered by the submitter. Where the number of submitter-entered terms was greater than the maximum available in the standardized list (seven terms), we used the result for seven. The results, illustrated in Figure 4, show that while the maximum achieved with any combination of terms is the best performing result (highest or equal highest in 100% of cases), the score achieved with the real HPO terms outperforms the equivalent number of standardized terms in the majority of cases (78%). This is a useful check on the use of simulated annotations, as discussed below.

Repeating the analysis using the parent and grandparent terms of our originally selected terms showed that the mean Exomiser combined score increased as we moved up in the hierarchy, that is, the broader terms (lower information content) resulted in a higher score for the confirmed causative variant. The mean score across the entire cohort increased from 0.48 to 0.71 to 0.78 (original terms to parent to grandparent terms). This held true across all numbers of HPO annotations and when evaluating both the entire cohort and the subcohort comprised of known CMS genes (Table 4b). However, it was not reflected in the mean rank of the causative variant, which showed a global worsening from original to parent to grandparent across all annotation numbers. When assessed with the known gene subcohort, our original set of terms had shown a clear trend of improvement in ranking as number of annotations increased, as described above. This trend was absent in the analysis of the mean ranks with the parent terms, while with the grandparent terms the trend was reversed and showed a slight worsening in mean rank with increasing annotation levels. Evaluating the scores of the second-ranked variants in the cohort where the known causative variant was consistently ranked top showed that the combined score of the second-ranked variant also consistently increased from original to parent to grandparent term use, thus showing that the use of less specific terms increases the scores of the incorrect variant as well as the correct one.

3.4 Genes not previously associated with CMS and inclusion of model organism and interactome comparisons (experiment 2b)

The mean rank of the variant-only score for the genes not previously associated with CMS (new CMS genes) was 4.25. When assessing these genes with the human data PhenIX filter, the rank did improve on addition of HPO terms (Table 4a), but not to the same levels as with the known genes. The improvement was due to the contribution of the variants in CTNND2, DHCR7, POLG, SLC25A1, and TOR1A, all of which do have a different human disease associated (Table 1) and some moderate overlap in phenotype with CMS. It is the cases that have not previously been associated with the human disease under examination where one might expect the alternative Exomiser algorithm, hiPHIVE (which combines human, animal and interactome data) to be of interest, since the effects of gene defects are often observed and published in animal models before a human disease analogue is found, and understanding gene function on the basis of relationships with other genes in related pathways is a common approach. We compared the Exomiser combined score obtained with the PhenIX algorithm with that obtained from the hiPHIVE algorithm and stratified the results by known versus new CMS gene. Our results show that mean scores from the known CMS genes increased from 0.627 to 0.944 when moving from PhenIX to hiPHIVE, while those from the non-CMS genes increased from 0.161 to 0.688. All cases showed this increase individually, and cases that scored in the lower range with PhenIX tended to increase more substantially with hiPHIVE, resulting in a convergence of scores at the higher range (Figure 5). This indicates that including animal and interactome data does increase the level of evidence about a particular gene:phenotype correlation for both known and novel genes. However, an important caveat is that we did not observe a corresponding improvement in rank of the known causative variant, because the score of other variants in the analysis was also increased with the addition of the animal and interactome data. Across all experiments and all variants included in the ranking (i.e., every variant considered and not purely the causative variant), the mean Exomiser combined score was 0.0389 with PhenIX and 0.334 with hiPHIVE—a tenfold increase. Thus we see that the Exomiser combined score increases across the board when moving from the PhenIX to the hiPHIVE algorithm, so it is important not to draw conclusions purely from the increase.

To find the reasons for the cases in which the algorithms did not rank our known gene in first position, we need to look at the individual cases and examine not only the score of the causative gene but also the other variants present. Looking at the individual genes, we see that COL13A1 is one where the addition of the animal and interactome data improved the performance, as the rank improved from 4 (PhenIX) to 1 (hiPHIVE). This gene is in fact a recent CMS gene, (Logan et al., 2015) now listed in OMIM and with new cases just published (Dusl et al., 2019). However, when analyzed against human data alone the Exomiser gene:pheno score was zero, thus it is evidently too recent to have been included in the gene:phenotype correlation used by Exomiser at the time of running the analysis, while the data from animal models and interactome data apparently makes the variant much more convincing, resulting in a score increase from 0.000849 to 0.843. In contrast, the rank of TENM2 worsened from 4 to 10, despite its absolute score increasing from 0.000812 to 0.194. The top scoring gene in this case, RFC3, received a score of 0.757. Clearly, then, even animal model and interactome data does not provide a high level of evidence for this gene as a cause of the phenotype. The third gene to show a change, SLC25A1, was ranked first by the PhenIX algorithm but second by the hiPHIVE algorithm. With hiPHIVE, the scores of the top three variants cluster closely together at 0.749, 0.747, and 0.746, indicating that the algorithm has very similar levels of evidence for each. Like COL13A1, SLC25A1 is a relatively recent CMS gene (Chaouch et al., 2014) and the evidence in databases is lacking. However, when annotations are added, Exomiser's performance does improve correspondingly. We ascertained that COL13A1 annotations had been added to the HPO gene:phenotype annotations after we had performed our initial experiments and therefore chose to re-run the experiment for COL13A1 with seven HPO terms and the PhenIX algorithm using the February, 2019 annotation set. This resulted in a highly convincing rank of 1 and combined score of 0.998 for our known causative variant. This chance opportunity to test the effect of the addition of highly relevant annotations confirms the ability of the Exomiser algorithm to respond when the relevant data is available to it.