On the cover: This cover image, by Qifang Xu et al., is based on the Special Article Benchmarking predictions of allostery in liver pyruvate kinase in CAGI4, Pages 1123–1131. DOI: 10.1002/humu.23222.

The Critical Assessment of Genome Interpretation (CAGI) experiment is aimed to define the state of art of genotype-phenotype interpretation. Here, we present the assessment of the CAGI p16INK4a challenge. Participants were asked to predict the effect on cellular proliferation of ten variants for the p16INK4a tumor suppressor, a kinase inhibitor coded by the CDKN2A gene. Twenty-two pathogenicity predictors were validated in terms of accuracy and reliability. Different assessment measures were combined in an overall ranking to provide robust results.

Competitive Growth Scores of UBE2I Mutations. A) A histogram of scores. Bars are colored in gradient from red (deleterious mutations) through white (wild type) to blue (advantageous mutations). The same color gradient is applied to corresponding residues in surface representations of the UBE2I structure for B) deleterious and C) advantageous mutations. The consensus substrate tetrapeptide PsiKxE (magenta) and the C-terminal SUMO peptide (green) are shown and select residues near the active site are labeled.

(A) Representation of the SNPs&GO Support Vector Machine-based algorithm. (B) Main input features in SNPs&GO. (C) Performance of SNPs&GO on 38,460 single amino acid variants (Capriotti et al., BMC Genomics, 2013). Evaluation measures (Q2, TPR, PPV, TNR, NPV, MCC and AUC) are defined in Supplementary Materials.

The Evolutionary Action equation is a first order perturbation form of the genotype-phenotype relationship, which predicts the action of genetic variants on fitness. This analytical approach is general and its terms can be numerically estimated from sequence data. Thus, in contrast to the current state-of-the-art methods, it requires no training over large datasets and numerous features. The objective assessments of the Critical Assessment of Genome Interpretation experiments consistently ranked Evolutionary Action one of the most accurate predictors.

We have participated in various challenges in the four CAGI experiments. We used predictions from PON-P and PON-P2 and additional details to develop models for addressing the challenges. Here, we summarized our approaches and their performances in challenge assessments. We discussed the types of lessons we earned from CAGI, impacts of CAGI on our research and development of predictor with improved performance. We also make some suggestions for future experiments.

By participating in the Critical Assessment of Genome Interpretation, we demonstrate the direct applicability of missense variant pathogenicity predictors in the task of the prediction of real-valued impact on biochemical, molecular and cellular function, as measured in in vitro experiments. Our work suggests that when a large number of structural and functional features are integrated into a learning algorithm that outputs smooth score distributions, pathogenicity predictors can model the biology shared by both of these prediction tasks.

We report results obtained using newly-developed ensemble methods to address two CAGI4 challenges requiring prediction of continuous protein activity: population rare variants in NAGLU that are representative of clinical encounters, and mutations from high throughput mutational scans on UBE2I. The methods are effective, ranking 2^nd for SUMO-ligase and 3^rd for NAGLU. The ensemble approaches also show considerable potential for estimating the reliability of pathogenic assignments, and are already reliable enough for use with a subset of mutations.

Homotetramer of human liver pyruvate kinase (PYK) with bound substrate (phosphoenol pyruvate), inhibiting allosteric effector (alanine), and activating allosteric effector (fructose-1,6-bisphosphate), compiled from multiple crystal structures of PYK.

Residue positions at which side chain removal alters allosteric coupling for Fru-1,6-BP activation (Q_ax-Fru-1,6-BP).

A simplistic allosteric mechanism model based on allosterically-relevant plasticity. This model was developed with acknowledged insufficient structural insights, but provided a framework for mutational probings of the allosteric binding site. Although we provide data that is not consistent with this model, the responses of many mutant proteins could be viewed as consistent, raising questions about how mutational probings should be used in studies of allosteric mechanisms.

The CAGI-4 Hopkins clinical panel challenge was an attempt to assess state-of-the-art methods for clinical phenotype prediction from DNA sequence. Participants were provided with exonic sequences of 83 genes for 106 patients from the Johns Hopkins DNA Diagnostic Laboratory. Five groups participated in the challenge, predicting both the probability that each patient had each of the 14 possible classes of disease, as well as causal variants. We discuss the accuracy of the predictions and their implications for further methods development.

The CAGI4 SickKids clinical genome challenge provided an opportunity to assess the landscape of genetic variants found in a difficult set of 25 unsolved rare disease cases. We used a three-stage pipeline consisting of careful analysis of data quality, classification of disease relevant gene-specific variants into seven categories, and prioritization of variants by compatibility with the complex phenotype of a patient.We were able to determine the genetic cause of a case of epileptic encephalopathy, a missense mutation in KCNB1.

The Critical Assessment of Genome Interpretation (CAGI) is a community experiment consisting of genotype-phenotype prediction challenges; participants build models, undergo assessment, and share key findings. For CAGI 4, three challenges involved using exome sequencing data: Crohn's disease, bipolar disorder, and warfarin dosing. We discuss the range of techniques used for phenotype prediction, the methods used for assessing predictive models, and the lessons gleaned from the CAGI exomes challenges.

We describe the design and implementation of a gene panel sequencing data analysis pipeline, VarP. The performance of the pipeline was assessed in the CAGI 4 community experiment. VarP identified the correct disease class and potentially causative variant(s) in 36/106 patients, including 10 patients where the clinical pipeline did not find any causative variants. Post analysis showed that use of three-dimensional structure could have assisted interpretation in a number of cases.

We designed an end-to-end deep learning architecture (called DeepBipolar) to predict bipolar disorder based on limited genomic data. Leveraging Deep Convolutional Neural Network (DCNN), it automatically extracts features from genotype information to predict the bipolar phenotype. DeepBipolar was considered the most successful by the independent assessor in the Critical Assessment of Genome Interpretation (CAGI) bipolar disorder challenge.

Understanding the basis of complex trait disease is a fundamental problem in human genetics. The three rounds of CAGI Crohn's Exome challenges provide insight into the adequacy of current disease models by requiring participants to identify which of a set of individuals has been diagnosed with the disease, given exome data. For the CAGI4 round, we developed a method that uses machine learning methods to assign disease risk, with only the imputed genotypes of GWAS marker SNPs as input.

This study proposed to perform complex trait prediction using a stratified design. The genetic data are divided into strata according to genetic architectures, and feature selection is conducted within each strata through a data adaptive W-test for main effect and pairwise interactions. An ensemble classification algorithm can be applied to integrate the selected features to perform prediction. Application on CAGI data set showed that including interaction effect of common variants improved prediction accuracy.

Deciphering the functionality of the non-coding genome has been the focus of many recent studies, aiming to annotate regulatory regions and understand their specific role in disease and other phenotypes. The goal of the CAGI eQTL challenge was to predict the activity of candidate genomic regions (experimentally evaluated by massively parallel reporter assays). Our meta-analysis of competing submissions highlights features and models that lead to accurate prediction and points to areas for improvement.

DNA variation in regulatory regions (e.g. enhancers) has been implicated as contributing to many human diseases. We have been developing quantitative computational models of regulatory regions to predict the impact of this variation and to identify the mechanisms of enhancer modulation. In this blind test, our model predictions are in quantitative agreement with direct measurements of enhancer activity.

We present a novel ensemble-based computational framework, EnsembleExpr, that achieved the best performance in the Fourth Critical Assessment of Genome Interpretation expression quantitative trait locus “(eQTL)-causal SNPs” challenge for identifying eQTLs and prioritizing their gene expression effects. We envision EnsembleExpr to be a resource to help interpret noncoding regulatory variants and prioritize disease-associated mutations for downstream validation.

PGP provides unrestricted access to genomes of individuals and their associated phenotypes. The CAGI PGP challenge is to predict whether an individual had a particular trait or phenotype profile based on their whole genome. Assessment results show prediction of individual traits is difficult, relying on a strong knowledge of trait frequency within general population, while matching genomes to trait profiles relies heavily upon a small number of common traits.