2.1 Modeling outline

The outline of our modeling workflow in CASP14 is shown in Figure 1. During the initial step, for every target we attempted to identify multimeric templates for comparative (homology) modeling using sequence- and structure-based search methods and to generate models of the whole protein complex. If we could identify templates, but the sequence-structure alignments were not reliable, we then used template-based docking, that is, we aligned monomer models to the chains of the multimeric template to obtain a multichain structure. Coiled-coil protein structures were predicted using a custom-designed procedure. If no templates could be identified, we switched to free docking. In the case of large multisubunit targets, we combined all the methods, employing a comparative modeling approach for the parts of the complex for which templates were available followed by docking of the resulting subcomplexes to obtain the full assembly. Compared to CASP13,¹⁸ the major modifications of our modeling pipeline included the introduction of structure-based searches for multimeric templates, the use of a novel model selection protocol emphasizing the interface scores, and the application of short molecular dynamics simulations to rigid-body docking models in order to improve the geometry of interchain interactions.

2.2 Comparative modeling

The outline of multimeric comparative modeling pipeline is provided in Supplementary Figure S1. Starting with target sequences, we first searched for potential templates using PPI3D¹⁹ and HHpred^{20, 21} web servers. If this step failed to identify reliable templates (HHpred probability ≥90%) and/or produced incomplete alignments, we additionally employed structure-based searches. In this case, selected monomeric CASP server models corresponding to the subunits of a target protein complex were used as queries for PDB searches using the DALI server.²² The aim was to identify multimeric PDB assemblies that could potentially serve as multimeric templates. Once templates were identified using either sequence- or structure-based searches, structural models for the whole protein complex were generated by MODELLER²³ and its plugin AltMod²⁴ using the multichain modeling function. For the simpler cases, where the use of PPI3D server sufficed, comparative modeling was automatic, except for the choice of structural templates and for the assessment and ranking of models based on different templates.

2.3 Template-based docking

For some targets neither sequence-based nor structure-based searches produced alignments with multimeric homologs of sufficient reliability and/or coverage such that these alignments could be used for multimeric comparative modeling. However, in some cases we observed that the proteins identified as potential multimeric templates had similar functional annotations as the CASP targets. If the alignment of CASP server models to these multimeric structures using TM-align²⁵ seemed plausible, we constructed assembly models by simply aligning monomeric CASP server models to different chains of the templates and then relaxing the resulting models to remove steric clashes using the same methods as in the case of free docking.

2.4 Coiled-coil modeling

Several targets were predicted to be coiled coils using MultiCoil2²⁶ and similar sequence analysis tools provided in MPI Bioinformatics Toolkit.²⁰ Such targets were modeled using a custom-designed procedure: structure models were automatically generated by MODELLER/AltMod using the same manually selected coiled-coil template and automatically generated all possible gapless target-template alignments followed by model selection.

2.5 Free docking

When no templates could be found for protein complexes, free docking of top five selected monomeric CASP server models was done by Hex²⁷ for hetero-complexes and Sam²⁸ for homomultimers (Supplementary Figure S2). The resulting models were ranked using both global and interface VoroMQA scores as described below. Next, top 100-500 models were relaxed by a very short molecular dynamics simulation using OpenMM software,²⁹ Amber99SB force field, and GBSA-OBC solvation model.^{30, 31} The relaxed models were subsequently re-ranked using the same scoring procedure and clustered according to the interface Contact Area Difference score (CAD-score)^{32, 33} values aiming to select a diverse set of models. The free docking workflow was fully automated, but the final models were always inspected visually.

2.6 Hybrid modeling

For some large target protein complexes structural templates were available only for some of the subunits or domains. In these cases, a hybrid modeling strategy was used, that is, part of the complex was modeled using comparative modeling, whereas remaining subunits were docked to it either by template-based docking using TM-align²⁵ or by free docking. In addition to that, a mixture of free docking and template-based models was submitted for several smaller targets that had templates only from structure-based search.

2.7 Model selection

For model selection we used both global structure scores and interface scores. This approach was described previously^{16, 18} and implemented in the VoroMQA web server,³⁴ but for CASP14 we introduced some modifications. These included VoroMQA-dark, a new method for global structure evaluation (see below), and an improved tournament-based ranking algorithm (see Supplementary Information for details). When ranking models based on their pairwise comparisons, the algorithm puts more emphasis on the interface pseudo-energy and less emphasis on the global structure score. This is achieved by using a tolerance value when comparing global VoroMQA-dark scores. If the difference between global scores is small, the interface scoring becomes the only decisive factor. We named the new VoroMQA-dark-based model selection protocol as “VoroMQA-select-new.” In addition to fully automated scoring methods, models were also evaluated according to constraints obtained from the literature or from the CASP contact prediction servers, if such data were available. All models were visually inspected before submission and manual ranking adjustments were introduced, if necessary. These manual adjustments were predominantly applied in the hybrid modeling cases.

2.8 VoroMQA-dark method for model quality assessment

VoroMQA-dark is a new model quality assessment method based on the previously published VoroMQA³⁵ method (which will be referred to as VoroMQA-light). VoroMQA-dark uses a neural network (NN) trained to predict local (per-residue) CAD-score³² values. The global structure score is computed by averaging the predicted local scores. The NN input vector for each residue is computed from the Voronoi tessellation-based contact areas and the corresponding contact potential values from VoroMQA-light. See Supplementary Information for more details on VoroMQA-dark. The VoroMQA-dark standalone software is included in the extension of the Voronota³⁶ package freely available from https://kliment-olechnovic.github.io/voronota/expansion_js/.

3.1 Overview of the results

To analyze our performance in CASP14 we used several accuracy measures designed to evaluate various features of multimeric models. For the overall model evaluation we used QS-score, a distance-based measure of interface accuracy.³⁷ To make qualitative model accuracy assignments we converted QS-score values to CAPRI-like accuracy categories.³⁸ Four other scores were used to assess the interface and the overall structure accuracy. Interface Contact Similarity (ICS or F1-score) and Interface Patch Similarity (IPS or Jaccard coefficient) were used to assess contact and interface patch prediction, respectively.¹⁴ Oligomeric lDDT and TM-score were used to assess overall structure accuracy. lDDT is an all-atom superposition-free score,³⁹ whereas TM-score is based on the rigid body superposition of Cα atoms.^{40, 41} In addition to the above scores reported by the Prediction Center, we also used CAD-score^{32, 33} to evaluate both structure and interface accuracy.

The summary of our modeling results based on the QS-score is presented in Table 1, whereas the detailed accuracy evaluation of our best models is provided in Supplementary Table S1. As can be seen in Table 1, for 11 targets we identified multimeric templates and used comparative modeling, producing medium or high-accuracy models for eight of them. In the absence of reliable target-template sequence-structure alignments, we applied template-based docking using TM-align. This approach resulted in models of medium accuracy for two targets. Hybrid approaches utilizing both comparative modeling and docking steps were used for nine targets with relative success. The results of free docking were ranging from completely incorrect to medium-accuracy models. The custom modeling procedure that we used for coiled coils did not produce any reliable models.

3.2 Modeling results in the context of other CASP14 groups and automated model selection

The results, presented in Table 1 and Supplementary Table S1, do not tell much about our relative success. To investigate our performance in the CASP14 context, we compared our results (group “Venclovas”) with those of three other top-performing groups for models designated as first (model 1). We also included our automatic model selection protocol (“VoroMQA-select-new”) as a virtual group, allowing it to make selections from all CASP14 multimeric models (produced by both automatic servers and human groups). By doing this we aimed to test the effectiveness of our automatic scoring in a scenario where a set of diverse models, generated by multiple methods, is available to select from. For the performance comparison, we used the sum of z scores of two interface accuracy measures (ICS and IPS) and two global structure accuracy measures (lDDT and TM-score).

The comparison, shown in Figure 2, revealed that different features of our models were predicted with different level of success. According to the accuracy of intersubunit interfaces (ICS and IPS) we achieved the best results. We were particularly successful in predicting interface patches (IPS). On the other hand, the global structure accuracy of our models is not so great compared to other top-performing groups. This is especially visible if we consider lDDT, an all-atom score, largely reflecting the accuracy of individual subunits. Interestingly, our automatic model selection protocol showed relatively strong performance, taking the third position by any of the four scores. Although this method performed worse than our human group on both interface accuracy measures and TM-score, the results according to all-atom accuracy (lDDT) were quite a bit better.

To look at different features in more detail, we examined per-target z scores. z Score values were accumulated progressively for targets ordered by the maximum ICS value of all the models produced by any group for a given target. Such an ordering may be interpreted as an estimate of the target difficulty. Figure 3 shows the resulting plots for the models designated as first (model 1). In addition to the data for the same top groups and “VoroMQA-select-new,” the plots also include the data for the best models provided by any predictor group. The latter curve may be considered as a reference by representing the upper limit of what could have been achieved in CASP14.

Interestingly, the per-target analysis (Figure 3) revealed that the relative success of different groups was dependent not only on the evaluation measure as seen in Figure 2, but also on the set of prediction targets. According to the interface prediction accuracy, our group dominated for most of the targets [Figure 3(A,B)]. On the other hand, if we consider the global accuracy of models the picture is different. According to TM-score [Figure 3(D)] our models are below the state-of-the-art for about half of targets, whereas according to lDDT [Figure 3(C)] this is true for nearly all the targets. To see whether our models as assessed by lDDT were indeed significantly inferior to those of other top groups, we examined the cumulative raw values (Supplementary Figure S3). Surprisingly, it turned out that the absolute differences between the groups, especially if evaluated using lDDT (Figure S3F), are relatively small. This indicates that in most cases subunit structures were of comparable accuracy and that relatively large z score differences resulted from small structural improvements (see examples in Figure S4). The same analysis performed with the CAD-score-based analogs of ICS, IPS, and lDDT scores led to similar conclusions (Supplementary Figure S5).

In addition to individual scores, we analyzed their combinations reflecting either the interface prediction accuracy or the accuracy of both the interface and the global structure. We performed this analysis both for models designated as first (Figure S6) and for the best-of-five models (Figure S7). The analysis of these combinations has further corroborated above observations on our relative success in the interface prediction and on target-dependent group performance. Interestingly, in the analysis of best-of-five models our automatic selection protocol (VoroMQA-select-new) was the best according to the interface accuracy [Figure S7(A,C)] and close to the top according to the combined accuracy [Figure S7(B,D)]. Although having access to all the models VoroMQA-select-new had an important advantage over other groups, the results suggest that this automatic selection procedure is quite robust.

3.3 Template-based modeling

As the structures of protein complexes are often evolutionary conserved,⁹ template-based modeling is currently the most reliable method to model them. Straightforward multimeric comparative modeling resulted in medium to high accuracy models for 8 of 11 CASP14 targets. Template-based docking also resulted in medium-accuracy models for two targets. Thus, if reliable templates were available the template-based approach worked well for both homomers and heteromers.

Identifying the correct template having the same oligomeric state is the key to successful modeling of protein complexes.^{16-18, 37} Ambiguous oligomeric state of templates may be the reason why we failed to model T1034, for which we used templates having different oligomeric states.

In CASP14 we had additional examples demonstrating the limitations of template-based modeling for protein complexes. One such example, H1036, represents a trimeric viral protein bound to an antibody. Our models were based on homologous trimer structures bound to antibodies. This resulted in good models of homotrimer interfaces, but the antibody was bound to a completely different epitope (Figure S8). This incorrectly predicted interface is not surprising bearing in mind the nature of antibody-antigen interactions. The binding site in the antibody (paratope) is formed by hypervariable loop regions, and the antigen-binding site (epitope) can be anywhere on the protein surface.⁴²

Our results for T1099 show another limitation of template-based approach for protein complexes. This target is a large viral capsid, yet modeling its structure can be reduced to a problem of predicting a homotetramer (T1099v0) having two different interfaces (T1099v1 and T1099v2). Our models contained high-accuracy interface 2, yet the interface 1 was incorrect (Figure S9). The reason for this failure was the large insertion in the target interface, compared to the template structures.

3.4 Hybrid modeling of large targets

When modeling a large protein complex templates might be available just for some parts of it. During CASP14 this was the case for heteromeric targets H1060 and H1097. For both of these targets more accurate models were generated for those parts that had templates.

H1060 was a viral protein complex containing 27 subunits (five homomeric rings bound to each other, A6B3C12D6, Figure 4). We found templates for all the rings and generated models of medium accuracy for all of them using either comparative modeling or template-based docking methods. Next, we tried to do template-based docking of the ring models to each other using another viral template, but for such a large complex the docking was complicated. As a result, the accuracy of hetero-complex (H1060v1) model is lower than the accuracy of models for individual rings.

Similar situation was observed with hetero-pentamer H1097, where we tried to dock the domains of the fifth subunit to a homology model of a hetero-tetramer albeit with limited success (Supplementary Table S2).

3.5 Free docking

Our free docking results show similar trends as in previous CASP and CAPRI experiments (Figure 5; Supplementary Table S3).¹⁸ First, the IPS values sometimes (in 4 of 11 analyzed cases) are much higher than ICS. This indicates that the residues mediating protein-protein binding are predicted better than the mutual subunit orientation defining the exact contacts at the interface. Another observation is that the accuracy of subunits matters a lot when docking modeled protein structures. We did not produce any acceptable accuracy models when we were starting from subunit structures of lower accuracy (lDDT <0.4 or TM-score <0.5). It is also important to note that the opposite is not necessarily true. Accurate models of individual subunits do not guarantee accurate docking models.

The most successful docking results were obtained for H1081v0, T1083 (Figure 6), and T1084. Interestingly, in the cases of T1083 and T1084 the free docking models were better than the template-based models, but the reasons for this are not clear. H1081 was a large target, where two decameric rings had to be docked, and for that we developed a custom procedure. The homology models of decameric rings were aligned on the axis perpendicular to the ring plane and then pushed to each other (using 1 Å steps) and rotated around the axis (using 2° steps), saving every distinct arrangement. Afterward all obtained models were relaxed, scored, and ranked. This custom “two-ring docking” procedure resulted in surprisingly good models.

The modeling of other docking targets was less successful (Figure 7) illustrating common problems related to the monomer model accuracy and scoring. For example, in the monomer structure of T1054 which we used for docking, the position of N-terminal helix is not compatible with the dimeric structure. The helix is too well packed against the subunit structure, occupying the place of a helix from another subunit in the dimer [Figure 7(A,B)]. Therefore, it was impossible to obtain a correct docking model starting from such a monomer. Interestingly, when during the post-CASP analysis we repeated the docking experiment using the same monomer model, but without the N-terminal helix, the docking was highly successful [Figure 7(C)]. Of note, in solution this protein exists as a mixture of a monomer and a decamer, and the dimer observed in the crystal might represent an intermediate state in decamerization.⁴³

In our best model for heterodimeric target H1065 one of the subunits is rotated ~180° compared to the experimental structure [Figure 7(E)]. Again, the interface patch is identified correctly while the interface contacts and subunit orientation are different. Both monomer models are fairly accurate (lDDT >0.65, CAD-score >0.7, TM-score >0.8), therefore, their accuracy probably is not the reason for incorrect docking [Figure 7(D)]. However, scoring is really problematic for H1065: both the global and the interface VoroMQA scores of the experimental structure and the model are highly similar (global scores: 0.70 vs 0.68, interface energy: −354 vs −388 for target and model, respectively). In other words, even if the experimental structure was present among the models it would not necessarily have been selected.