2.1 Problem definition and model overview

In TCR–epitope binding identification task, the aim is to predict whether the given TCR and epitope bind to each other. The epitope information is given as a short peptide sequence which consists of several amino acid residues. For TCRs which compose of α and β chains as well as other extra information (e.g., V, D, and J genes), the complementarity-determining region 3 (CDR3) region of β chain is the most representative element for TCR since this region typically locates most closely to corresponding epitope. One of 21 letters is used to represent each amino acid residue of both types of sequences. Given TCR and epitope sequence as input, the model output a continuous value between 0 and 1 that indicates the probability of binding.

The proposed framework called TEPCAM is an end-to-end model (Figure 1). The raw epitope sequence and aligned TCR sequence are encoded by an embedding block with additional positional encoding strategy. Then the encoded sequences go through the attention module which contains a self-attention layer and a cross-attention layer, in which the multihead attention mechanism is often used to process contexture information. While self-attention layer obtains the query, key, and value from single sequence, cross-attention transforms one input sequence as query and another sequence as key and value respectively, which is a better way to capture more specific interactions between two parts of input by allowing model to attend to both of sequences simultaneously, leading to better generalizability. The output of attention module matrix goes through a multi-channel convolutional layer in order to further process features, which are entered into final feed-forward layer consisting of three fully connected layers with batch normalization and GeLU activation function to output a final value.

2.2 TEPCAM achieves better generalizability than several state-of-the-art models

The experimental validated datasets were used to train and test models. In general, we trained TEPCAM using TEP-merge dataset which were merged from three public databases, including VDJdb (Goncharov et al., 2022), McPAS and IEDB (Tickotsky et al., 2017; Vita et al., 2019; Figure 2A). To avoid external bias from the negative sample generation strategy (Moris et al., 2021), we did not use a background TCR dataset such as data from 10× Genomics assay or TCRdb but applied the strategy of randomly shuffle positive pairs to generate negative pairs (Chen et al., 2021; Montemurro et al., 2021). Finally, TEP-merge consists of 129,654 TCR–epitope pairs with a positive: negative ratio as 1:1, involving 1523 unique epitopes and 60,342 unique TCR. The imbalance and longtail of dataset were observed on the epitope side (Figures 2B,C, S1, and S2), in which the top 117 epitopes accounted for 90% of the dataset, while the remaining 1406 epitopes contributed to only 10% of the data. As for TCR, the longtail distribution is relatively less pronounced. Instead of random split dataset in which most of TCRs and epitopes in the test set are likely to be appeared in the training set, we strictly split the dataset to construct a zero-shot TCR prediction task to test model performance when generalized to novel TCR sequence which demands high generalizability. In this setting, the TCR sequences of test set were unseen in the training set.

To verify our model's generalizability, we first compare TEPCAM against other four TCR–epitope binding specificity prediction models on the strictly split TEP-merged dataset. Four baselines are all deep learning-based models, which were established in a supervised way, including TITAN, ERGO-AE, ERGO-LSTM, and ATM-TCR. The dataset was split into a ratio of 4:1 for training data and test data. The split was repeated by 10 times in order to minimize the impact of randomness.

In this TCR zero-shot setting, TEPCAM showed better performance than other four state-of-the-art methods (Figure 3A and Table S5). The higher values in terms of Accuracy, AUC, AUPRC, and F₁ were observed. Moreover, we implemented the test for 25 times by splitting dataset using 5 different random seeds. For each random seeds, we applied five-fold cross-validation (CV) that every fold contained unique TCR, and then we calculated the mean value and variation for each metric. Our model also showed low variation values among the metrics of Accuracy, AUROC, and AUPRC. The results indicated that TEPCAM could successfully generalize to the unseen TCR sequences.

Then we applied TEPCAM to an external dataset named ImmuneCODE in order to test the performance when transferring to another dataset with potentially different distribution. After the same filtering, the final ImmuneCODE dataset contained 28,303 TCR–epitope pairs assigned to ~1000 Coronavirus disease 2019 (COVID-19)-related individuals (Nolan et al., 2020), in which the pairs with TCR appeared in merged dataset were removed. All the models were trained using the entire TEP-merge dataset, and then tested in ImmuneCODE. As shown in Figure 3B and Table S6, TEPCAM outperforms other models in terms of all the metrics on this external dataset. Notably, in comparison to the training and test data sourced from one dataset, the external test is of more challenging, since the distribution of external dataset may be more distinct, and the significant distribution shift can make it more difficult to generalize the model on unseen data. In sum, experimental results demonstrated that our model exhibits superior generalizability compared to other recently published frameworks.

2.3 Detail analysis of predictive power of individual components in the model

In this subsection, we further assessed the predictive power of individual module of the whole model. The advantages of TEPCAM can be summarized as two parts. First, in a known binding epitope–TCR pair, the TCR recognizes epitope by only using several key residues, and similarly, the epitope also interacts with TCR by using only a few specific residues. The cross-attention module of our model is capable of capturing the specific residue-wise interaction between two input sequences. Second, the output of cross-attention encoded the interaction map of TCR and epitope sequences and such matrix is suitable to be further processed by convolution neural network for feature extraction and learning local pattern. The CNN layer we used did not modify the matrix shape but increased the channel number, enabling the model to enrich more knowledge. We conducted a series of ablation analysis on the ImmuneCODE dataset. First, a drop of performance was observed when either the self-attention module, cross-attention module, or convolution module was removed (Table 1), but the metrics are still higher than that of the baseline models. This result may be attributed to the fact that the attention layer after embedding module can extract useful features for prediction, although one of them was removed. Then we performed a module swap by repositioning the cross-attention module after the convolution module. As a result of this rearrangement, we observed a decrease in the model's performance.

Before going through the attention module, the features were initiated originally from an embedding block. We also tested the different strategies to get sequence embeddings, including BLOSUM62 matrix and two pretrained embedding models. BLOSUM62 matrix naturally contained the relationship between one residue to another by mapping one amino acid to a 24-dimensional space (Henikoff & Henikoff, 1992). In our settings, a lower AUROC value of 0.666 was observed for BLOSUM 62 matrix in comparison with 0.679 for encoding sequence by our embedding layer (e.g., random initialization). Then, we employed pretrain model originally such as TCR-BERT and TCR2Vec, which are two pretrained frameworks published recently, training on a large TCR sequence pool. TCR-BERT embedded the sequence into a 768-dimensional vector and TCR2Vec set the hidden size as 120 (Wu et al., 2021; Jiang, Huo, Zhang, et al., 2023). Both TCR-BERT and TCR2Vec encoded the sequence to a fixed length, and the output was presented as a high-dimension vector that contained enriched evolutionary information, and improved performances were observed on TCR–epitope binding classification tasks comparing to general protein language models such as TAPE and ESM series. We tried to apply these two models in the embedding module. However, the implementation of TCR-BERT and TCR2Vec even decreased the model performance (Figure 4A).

Deep models sometimes tend to get better performance on majority samples. For TCR–epitope-related dataset, the diversity of TCRs is far more than that of epitopes, and the distribution of epitopes is severely imbalanced. We investigated the performance of per epitope in both of two tasks after removing the epitopes that are assigned by <20 TCRs to avoid extreme values that skew the model performance. For task1, the epitope counts and metrics were shown in very weak negative correlations, suggesting that our model is not sensitive to epitope distribution on this dataset (Figures 4B and S3–S5). On the ImmuneCODE test set, positive correlations were observed. However, when we further searched the top 20 epitopes in terms of AUC which had the best prediction performance (Table 2), the epitope counts and metrics had negative correlations, which indicated that the counts for those epitopes identified accurately did not affect model performance. (Figure 4C,D). These findings supported that our model was not affected too much by the inner bias of training data, suggesting that the TEPCAM model could learned some binding pattern that determines the binding of two sequences.

To further illustrate the advantages of our model, we then evaluated the model performance on the level of specific epitope. First, we benchmarked TEPCAM against other four baselines. The performance on individual epitope was evaluated using AUROC as metric, and then the number of epitopes that obtain Top 1 AUROC for each model is counted. We found that TEPCAM achieved the highest Top 1 AUROC on both Task 1 (CV on merged dataset) and Task2 (benchmarks on external dataset ImmuneCODE; Figure 5A). Second, we ranked epitopes by their counts on training set. Then the most abundant 20 and fewest 20 were selected. We found ATM-TCR performed best among four baseline models, so we focus on the comparison between TEPCAM and ATM-TCR. TEPCAM performed better in most epitopes, especially a few bottom epitopes which TEPCAM obtained acceptable AUROC when ATM-TCR almost failed to determine (Figure 5B).

2.4 Attention map provides the clue for the learned binding patterns of TCR–epitopes

TEPCAM has an advantage on model interpretability that enables a closer look at the interaction patterns at residue level, which is critical for detecting specific biological rules. Basically, the interaction strength could be represented by attention score, which is calculated by the scaled dot-product of Key and Query. We extract the attention map in self-attention layer and cross-attention layer from ImmuneCODE test dataset to validate whether our model learned salient binding pattern. For both TCR and epitope, the self-attention maps were condensed to a one-dimensional vector by taking the mean across other dimensions (Figure 6A). The results depicted a higher attention score for the middle part of TCR sequence, and the scores on epitope side are close to uniform distribution except for beginning and final positions, suggesting that the self-attention layer mainly focuses on the middle part for input data, especially for TCR. Two-dimensional attention maps extracted from cross-attention reveal more detailed information about interaction patterns between TCRs and epitopes after averaging over all samples (Figure 6B). The matrix shows a higher score in the middle part of TCR toward epitopes, and the fine-gained interaction map could provide more biological insights on the specific cases.

2.5 Application of TEPCAM on the TCRs with 3D structures

We finally applied TEPCAM to a high-quality dataset from STCRdab (Leem et al., 2018), which records the curated TCRs with three-dimensional crystal structure data stored in Protein Data Bank (PDB). We selected TCR structures with assigned peptide epitope presented by MHC Class I molecule. Moreover, we performed epitope zero-shot task by removing overlapped epitopes in training data. In the scenario of unseen epitope task, the epitope zero-shot task refers to the recognition of TCR–epitope pair in the scenario that epitopes in test are unseen in the training set. Most of models failed to predict correctly and the accuracy drops to the random guess level. This task is particularly challenging, partly due to the very limited epitope diversity. We obtained 128 high-quality items with three-dimensional structure and extracted the corresponding epitope and CDR3 sequences for model test after removing overlap items. Since this test dataset only contains positive samples, we evaluated the performance of our model using the accuracy metric. TEPCAM shows a recall rate of 0.57, suggesting that more than half of the binding samples were successfully predicted. We also evaluated the model's performance in this scenario by generating negative samples using a random shuffle strategy. In this benchmark, TEPCAM achieved the highest F₁ value of 0.587 (Table S4). To prove our model not just successfully predict items with epitope close to training set, we explored the relationship between binding probability and epitope distance. Here, smallest levenshtein distance acted as distance for one epitope on high-quality dataset. Unexpectedly, the spearman correlation between levenshtein distance and probability was 0.286 with a p-value 0.001, indicating that the predicted probability of increases as epitope's levenshtein distance increases (Figure 7). These interesting results prove that our model could successfully retrieve a part of unseen epitopes, without relying on the presence of similar epitopes on the training set.

Using interaction map extracted from cross-attention layer enables us to easily investigate the region on which our model focuses. We picked TCR–epitope complexes that are recorded with PDB ID 2BNQ and 5EU6 as examples (Figure 8). As shown in Figure 8A, the glycine residue (G*5) on the epitope located in the binding surface located close to G*99-T*101 of TCR CDR3 region. The corresponding attention scores of G*5 to G*99 and G*5 and G*100 were observed highest in the cross-attention matrix, suggesting that our model had caught these pairs that have potential to determine the binding between TCR and epitope. The second case came from protein complex with PDB ID 5EU6 (Figure 8B). The tryptophan residue (W*5) located at the fifth position of the epitope exhibits a greater degree of attention compared with other positions. As shown in the crystal structure, this tryptophan residue has an aromatic ring toward the central of CDR3 loop region, resulting in a closer distance for tryptophan residue to several residues in TCR sequence. The distance between the aromatic ring to tyrosine (Y94), valine (V95), and glycine (G96) were among 5 Å. TEPCAM had captured two of three interaction pairs. These insights suggested that TEPCAM had learned some knowledge about the underlying binding mechanism.

5.1 Data curation

We collected TCR–epitope pairs from three databases as original source: VDJdb (Goncharov et al., 2022), McPAS (Tickotsky et al., 2017), and IEDB (Vita et al., 2019). For each database, we only selected the epitopes presented by human MHC I molecules. We only kept CDR3 region of β-chain in TCR sequence since this region contains most valuable information for recognizing epitope and the databases mainly record sequence information of β chain with very limited information about other chains (e.g., TCR alpha chain, V, D, and J genes). Length of sequences was filtered between 10 and 20 for TCR sequences and 8–11 for epitope sequences. In order to obtain a larger diversity of sequences, especially for epitopes, we merged three datasets and removed duplicated pairs. Finally, the merged dataset contains 64,827 positive pairs. To generate negative samples, we randomly selected a TCR sequence from our dataset except the paired one for an epitope, and obtained a mismatched TCR–epitope pair. To achieve a balance between the recall rate and precision, we constructed a balanced dataset with an equal ratio of positive to negative samples. This strategy is the same as related works such as TITAN, ATM-TCR, and ImRex. Following this way, we construct negative data with the same number as positive data. We finally got a dataset with 129,654 pairs, named TEP-merge.

To validate our model's generalizability, the data from ImmuneCODE was collected as a held-out dataset (Nolan et al., 2020). The quality control filters were the same as TEP-merge. Moreover, we removed pairs whose TCR presented in the TEP-merged dataset. The high-quality dataset was constructed from STCRDab (Leem et al., 2018), in which we attained 128 TCR–epitope pairs with three-dimensional structural information stored in PDB database.

5.2 Details of the model architecture

The concatenated matrix output of cross-attention layer is finally sent to convolution module, which composes of three layers of convolutional neural networks. Convolution neural network is widely applied in computer vision, which is also suitable for this task since the attention matrix can be regarded as a picture with pixels. We used a single convolution kernel with size = 3 and stride = 1. The padding number was also set to 1 in order to maintain the output size. The number of channels was increased gradually from 1 to 4, 16, and 32 for output, and the batch normalization was applied after convolution operation, following the activation function GeLU (Hendrycks & Gimpel, 2016). The features passed through CNN are averaged in the channel dimension and concatenated. A residue connection was applied for the flatten output of cross-attention. Finally, the concatenated features are sent into feed-forward layer with three fully connected layers. The perceptron has 1024, 128, 16, 2 neurons for each linear layer and the output also activated with GeLU. Finally, the Softmax was used to dense the output into a range from 0 and 1. By sequentially stacking these four core modules, the input sequences are processed from multiple perspectives and information is sufficiently exchanged to yield relatively reliable predictions. We use the cross-entropy loss as loss function.

5.3 Implementation and hyperparameter tuning

The data process was conducted by using Biopython 1.78, Numpy 1.23.5, and Pandas 1.5.2. We used PyMol version 2.5.5 to visualize protein three-dimensional structure. The visualization of results mainly used ggplot2 package in R. TEPCAM was trained on two GeForce RTX 3080 GPUs with 16G memory under the PyTorch 2.0.1 framework with Python 3.8.16. The model weights were updated using AdamW optimizer with 1e−4 weight decay.

We conducted hyperparameter tuning using random-split strategy (Note S1), and the optimal hyperparameters are: Number of head_{self-attention} = 3, Number of head_{cross-attention} = 6, dimensions of embedding and attention layer = 32, learning rate = 5e−4 (Table 3). We found that TEPCAM was relatively robust across different hyperparameter values. The results of hyperparameter tuning are given in Table S1–S3.

5.4 Evaluation metrics

5.5 Baseline models

To compare the generalizability of our model, four supervised model are selected as baselines, which are published recently and showed state-of-the-art performance on the public datasets. ERGO-LSTM and ERGO-AE are based on LSTM and Autoencoder (Louzoun, 2020). Since we only use CDR3-beta as TCR input, the next generation of ERGO that contains more knowledge was not considered. TITAN used bimodal attention mechanism and pretrained on a CPI task (Weber et al., 2021), and ATM-TCR applied multihead attention network to capture contexture information (Cai et al., 2022). We obtained the baseline models from their GitHub repositories and trained on our dataset for fair comparison. TCR2Vec and TCR-BERT are two pretrained models, and we used the weights of the existing model they released for comparison with our embedding strategy.