Assessing the Enrichment Performance in Targeted Resequencing Experiments^†

Introduction

Since next-generation sequencing technologies were launched just a few years ago, systematic resequencing of the human genome in larger cohhorts has found its way into small-sized and medium-sized research centers. As whole-genome sequencing is still comparatively expensive, target enrichment strategies are currently a very common technique in ongoing resequencing studies. In traditional Sanger sequencing, the target regions on the genome used to be enriched by PCR, which is not feasible in terms of cost and labor input when larger genomic regions are supposed to be selected. Microdroplet-based PCR [Tewhey et al., 2009] enrichment enables a larger panel of DNA target regions to be selected, although this technique is still far from being suitable for exome-scale studies. Higher throughput can be achieved, for example, through enrichment by hybridization on high-density microarrays [Albert et al., 2007] or in-solution hybridization [Gnirke et al., 2009], enabling whole-exome capture and a diverse degree of automation. Each of these technologies has its issues and technical peculiarities, which have been reviewed by Mamanova et al. (2010). Array-based and in-solution capture methods of the previous generation were compared in a recent study [Kiialainen et al., 2011].

As the performance of each of the aforementioned assays is also highly variable from sample to sample, enrichment parameters are very important quality criteria to be assessed after sequencing. On the other hand, there is no out-of-the-box software solution to this computationally intensive task, making people define their own standards and put efforts into the development of software, which often remains at an immature stage. To fill this gap, we have developed a methodological standard for the evaluation of the performance of target enrichment in next-generation sequencing experiments. We provide a software named NGSrich to apply our methods in a high-throughput fashion as a part of an integrated laboratory pipeline (Fig. 1). As the near future will also bring sequencers of the second generation into smaller laboratories, the rapid analysis of the enrichment performance for different samples and enrichment techniques also needs to become achievable without the intensive help of bioinformatics experts.

As an intriguing application of our software, we show the results of a comparison study of the quality and features of three enrichment products, which have recently been launched by leading companies and established in many sequencing laboratories around the globe (Fig. 2).

Materials and Methods

Statistical Framework

A target region specification is usually given as a browser extensible data (BED) file containing multiple lines or target regions per gene. The initial step of our approach to target enrichment evaluation is to scan this file for overlapping regions and merge any two of them into one contiguous target line. For a formal description, we first denote the total number of reads aligned to the genome by R and the number of reads covering a genomic position k by c_k. Considering a particular gene G represented by multiple target intervals G₁, …, G_n each of which typically comprises one exon, we denote the average coverage of a particular target region G_i by

and the average coverage of gene G by

To quantify the success of the enrichment procedure, we evaluate the percentage of target bases m ∈ G_j for which c_m exceeds a certain threshold and compare the mean coverage C(G_j) across the target regions. In this perspective, genes with very low or very high coverage, that is, those which are most affected by the unevenness of the enrichment procedure across the exome or target region, are of particular interest. Furthermore, to give evidence on the fraction of data that is usable at all, we calculate the percentage of reads that align to any target region G_j. As many on-target reads usually map across the target boundaries, we include the reads aligning 100 bp upstream or downstream of the target region.

For a target interval G_j, we define an enrichment factor as the ratio C(G_j)/E between the average coverage in the respective target region [C(G_j) as defined above] and the average coverage expected without any enrichment (E = LR/g, where L is the read length, and g is the genome size in base pairs). We assess the uniformity over a particular target region graphically by histogram plots.

To compare the performance of different samples and technologies, we propose calculating the number of reads aligning on the target region per kilobase of target sequence normalized by the total number of reads aligning to the genome, that is,

where R_t is the number of reads on target, and g_t is the length of the target region. As before, we include into R_t the reads aligning 100 bp upstream or downstream of the target regions into this statistic. Following the terminology of an approach to transcript quantification in RNA-Seq (Reads per Kilobase and Million mappable reads “RPKM”) proposed by Mortazavi et al. (2008), we usually call this ratio TPKM (on-Target reads Per Kilobase target region and Million mappable reads).

Integration into a Computational Pipeline

We developed the NGSrich software as a part of a computational pipeline used for high-throughput analysis of target-enriched resequencing experiments (Fig. 3). A compute cluster and the ELANDv2 software are used to run fully automated reference alignments operated by a relational database supplied with sample and species information. Variants called with CASAVA v1.8, SAMtools v0.1.17, and Dindel v1.01 [Albers et al., 2011] are also organized in this database and evaluated with further (yet unpublished) analysis scripts. Detailed reports on the target enrichment performance produced by NGSrich and reports from basic quality checks of the raw data using FastQC (available at http://www.bioinformatics.bbsrc.ac.uk/projects/fastqc/) are copied to a local web server. Using an efficient user management, these can be accessed by a predetermined group of partners defined by the pipeline operators. Our analysis system has been successfully used for processing of more than 400 exomes and targeted resequencing studies [e.g., Huebner et al., 2011].

Results

Initially, we used our statistical framework and the NGSrich software on a high-performance compute cluster to process data from one human exome enriched by Illumina TruSeq technology. From this, we got a browsable, HTML-based output including target level details for convenient use in a server-client environment (Fig. 1). We also successfully used the software for evaluation of customized microarray-based capture assays (unpublished data).

Subsequently, we employed our software to do a systematic evaluation of three human exome enrichment techniques. Our comparison study comprises 18 exomes prepared using (1) the Agilent SureSelect Human AllExon 50Mb kits (Agilent Technologies), (2) Roche NimbleGen SeqCap EZ Human ExomeLibrary v2.0 technology (Roche NimbleGen Inc.), and (3) the Illumina TruSeq Exome Enrichment Kit. Sequencing of the respective libraries on an Illumina HiSeq 2000 sequencing instrument resulted in between 109.8 and 225.0 million reads after Illumina signal purity filtering, whereas the number of reads that could be aligned to the hg19 reference genome ranged between 64.0 and 180.5 million (Table 1).

Regarding the enrichment performance (Table 1), the average target coverage ranged from 141.64 to 178.48 (Agilent Technologies), from 93.03 to 205.23 (Roche NimbleGen Inc.), and from 59.16 to 110.25 (Illumina Inc.). The analysis also shows that the target is covered at least 30× in between 61.32% and 85.63% (Agilent Technologies), between 88.52% and 92.95% (Roche NimbleGen Inc.), and between 75.26% and 83.97% (Illumina Inc.) of the total target region length. The latter statistics highly depend on the total amount of sequence data available and the target region specifications and size, which are provided by the respective vendor. To address the variability in the number of sequences, we repeated the evaluation with a random selection of 60 million reads from each sample, which essentially confirmed that the Roche NimbleGen product performs better than the other assays (Supp. Table S1). However, as the read count generated for resequencing actually needs to account for the target size, this analysis obviously favors a product with smaller regions for target capture.

On the other hand, the percentage of reads that align in the target regions or 100 bp upstream or downstream ranged between 57.72% and 70.95% (Agilent Technologies), between 75.92% and 79.01% (Roche NimbleGen Inc.), and between 69.61% and 71.24% (Illumina Inc.). Using the TPKM statistic for a normalized comparison between different samples and library preparation techniques (see Materials and Methods), the average and standard deviation of the values are 12.2 ± 1.12 (Agilent Technologies), 17.5 ± 0.27 (Roche NimbleGen Inc.), and 11.32 ± 0.11 (Illumina Inc.). A gene-level exploration of the data shows that between 141 and 143 (Agilent Technologies), between 115 and 135 (Roche NimbleGen Inc.), and between 44 and 72 (Illumina Inc.) are covered below 2× (Fig. 2). Moreover, 34 genes were poorly enriched (covered below 2×) in all six samples prepared with Illumina technology, 101 in those prepared with Roche NimbleGen technology, and 101 in those prepared with Agilent technique, again using average gene-wise coverage below 2× as a cutoff (Supp. Table S2). Taking together the three exome specifications, there are 20,925 RefSeq genes represented in total. Among these are 737 genes, which are not included in the Agilent target (but in the other two specifications), 1,737, which are not on the Roche NimbleGen, and 599, which were not on the Illumina target (Supp. Table S3). To exclude that the results are biased by low-quality alignments, which could be due to repetitiveness or structural peculiarities of the target sequences, we repeated our analysis using only those reads that have a mapping quality value of 30 or higher. We observed that the statistics are actually very robust against this (Supp. Table S4) and conclude that on a whole-exome level, the length of unmappable regions is rather small compared with the total length of the target sequences.

The exome coverage is often considered sufficient when it is 30× or higher in at least 80% of the target sequence. Our data allow the conclusion that the amount of sequence needed to achieve this is 14 Gb for the Agilent technology, 5 Gb for Roche NimbleGen, and 9 Gb for the Illumina protocol (Table 2), implying that in a multiplex library on one HiSeq 2000 lane, two (Agilent Technologies), five (Roche NimbleGen Inc.), or three (Illumina Inc.) human exomes can be processed, depending on the enrichment product that is being used.

To benchmark the results produced by our NGSrich software against a different approach, we generated a pileup version of the respective sample BAM files using the mpileup command of the SAMtools, version 0.1.17 [Li et al., 2009]. Subsequently, we evaluated the coverage information from the pileup file using basic statistical functions in R. Strikingly, this rather direct evaluation of coverage parameters matches the results produced by the software very closely (Table 3).

Discussion

A fair cross-platform comparison needs to take into account that the definition of the exome and the total size of the target regions are variable across platforms. We have, therefore, proposed to normalize the amount of on-target sequence to a per-kilobase value of the target reference. To address variability in sequence cluster densities and other issues, we have also proposed to normalize by the total amount of sequence and express both in a statistic called TPKM (see Materials and Methods). This statistic measures the amount of sequence over the specified target region just as the RPKM statistic does for a particular transcript [Mortazavi et al., 2008]. The larger the target region, the smaller the TPKM is for a fixed percentage of reads aligning on target. On the other hand, the larger the target region, the more on-target reads are needed to achieve the same TPKM value for a fixed total number of reads. Regarding the results produced by our software for the 18 exomes, it shows up that the TPKM value is very robust across samples enriched by the same technology and therefore suitable as a characteristic measure of a particular technique.

The observation that the TPKM values are clearly larger for the NimbleGen technology than for the other two approaches reflects in part that this protocol has the smallest total length of target regions. However, the percentage of on-target reads and other statistics also show that this approach actually performs better than the other techniques under consideration. In contrast, the smallest number of genes covered insufficiently (below 2×) is achieved by the Illumina TruSeq exome, although this technology has the largest total target region length. On the other hand, the per-sample time and cost needed for library preparation will always contribute to a decision for using a specific technology. Although the Agilent SureSelect assay (Agilent Technologies) has the poorest and least robust enrichment performance, it is comparatively fast and the handling and training for the workflow are easy especially for small projects. For the NimbleGen technology, the performance is superior to the other assays but this method has the longest preparation time and is therefore less flexible than the others. Also, the size of the target region and thus the gene content is smaller than for the other assays. The Illumina technology provides the easiest workflow for large-scale projects because the samples of a multiplex library can be pooled beforehand and, consequently, the enrichment process has to be run only once. Finally, the number of exomes that can be processed on one lane has direct impact on the sequencing cost, regardless of the price paid per gigabase.

The performance of the enrichment process also depends on the structure of the target sequence, for example, repetitiveness, GC content, and other issues. The general observation that a nonnegligible fraction of the genes covered insufficiently in a specific sample is poorly enriched also in most of the other samples processed by the same product (Supp. Table S2) suggests that failure of a specific subset of the target can be considered an assay-specific characteristic. Because the starting point of our software tool is the vendor-provided target specification file, our study can only measure the assay performances against the description of the vendors rather than using the exact coordinates of the hybridization targets. The way how target regions are declared is highly variable between the products. Regarding this, a reanalysis with respect to a reference exome downloaded from the current University of California at Santa Cruz (UCSC) annotation database showed that due to its more comprehensive capture regions, the Illumina TruSeq exome has the largest fraction of this reference exome covered to at least 30× coverage (Supp. Table S5), whereas a lot of genes is missed with the NimbleGen in-solution assay (Roche NimbleGen Inc.). This also suggests that the target definition is a very important point to consider when choosing a particular enrichment product. In addition, the way in which the performance depends on structural peculiarities of the underlying sequences could be subjected to further studies. The sequence capture methods reported here have recently been compared in several other studies all of which concluded that the Roche NimbleGen SeqCap (Roche NimbleGen Inc.) produces a higher percentage of reads, which can be aligned on target than the other assays do [Asan et al., 2011; Clark et al., 2011; Parla et al., 2011; and Solunen et al., 2011]. Clark et al. (2011) also mentioned that the Illumina platform is the only one to contain untranslated regions (UTRs) and hypothesized that the superior performance of the Roche NimbleGen assay (Roche NimbleGen Inc.) is due to its higher bait density.

Our NGSrich software meets the very common demand for a detailed, summarized, and exon-wise analysis of the target enrichment performance of next-generation sequencing libraries. Combined with a scriptable operation mode and HTML-based output reports for efficient integration into a web server, our software is capable of being used with extremely high throughput. With the publication of our software, we have closed an essential gap in the availability of software tools needed to build fast and efficient computational pipeline systems for resequencing experiments.