2.1 Data source

The generation of the current positive data set began by retrieving 3580 and 2852 virulent protein sequences from the VFDB (Liu et al., 2021) and UniProt (UniProt, 2021) databases, respectively (Figure 1).

2.2 Data pre-processing

The core dataset file was downloaded from VFDB, whereas we used virulence-related keywords such as virulence, adhesin, adhesions, toxin, invasion, capsule and so forth to obtain sequences from UniProt (UniProt, 2021). Further, these 6432 sequences were strictly screened to filter out the entries labeled as “probable,” putative, similarity, fragments, hypothetical, unknown, and possible (Figure 1).

The annotated sequences of bacterial enzymes were downloaded from the UniProt (UniProt, 2021) database to make a negative dataset. Here, the nonvirulent protein sequences were searched mainly in the bacterial proteomes from which virulent protein sequences were obtained for the positive dataset. This dataset was screened strictly to generate a high-quality negative set.

Subsequently, the sequences were clustered and compared using CD-HIT (Fu et al., 2012) at a cut-off value of 0.5 to remove identical sequences and reduce redundancy among positive and negative datasets. First, we obtained 3403 sequences out of 6432 using CD-HIT for the positive dataset. Further, to remove related lines among positive and negative sets, these datasets were combined and CD-HIT was reapplied with a cut-off value of 0.5, resulting in 3375 virulent protein sequences. Eventually, the processed dataset consisted of nonredundant 6781 sequences, comprising 3375 virulent and 3406 nonvirulent protein sequences (randomly selected to balance the positive dataset) to generate trained models (Figure 1). The distribution of virulent protein sequences from different bacterial pathogens after refinement is shown in Table S1.

2.3 Preparation of training and test datasets

Preparation of training and test datasets is for machine learning. In the present study, the positive and negative datasets were randomly shuffled and divided into 80% training and 20% test datasets. To ensure the absence of homologous proteins between these training and test datasets, we built a local database with training dataset sequences (both positive and negative dataset protein sequences, n = 5425), and performed a PSI-BLAST search (using PSI-BLAST run options “-e 0.001 -j 3 -m 9 -v 1 -b 1”, where e is E-value, j is the maximum number of passes, and the rest of the variables are related to PSI-BLAST output format) against this database with the test datasets protein sequences (n = 1356). From Table S2, only 6.05% (n = 82) of test dataset protein sequences showed similarity with training dataset proteins at an identity percentage value of ≥50. Table 1 provides the distribution of positive and negative dataset protein sequences used in the present study. Further, the training dataset was randomly divided into an actual training set (0.9 fraction of the data) and validation set (holdout 0.1 fraction of data) through an in-built feature of AutoGluon (Erickson et al., 2020), for the training and internal evaluation of machine learning (ML) models, respectively.

2.4 Calculation of protein sequence composition

The standalone package named GPSR (Saini et al., n.d.) was used for the calculation of the amino acid composition (AAC), dipeptide composition (DPC), and tripeptide composition (TPC) of virulent and nonvirulent protein sequences.

2.5 Calculation of protein sequences PSSM profiles

We used an in-house PERL script to calculate the PSI-BLAST (Altschul et al., 1997) generated PSSM profiles. A PSI-BLAST iterative search was performed against the SwissProt database with an E-value cut-off 0.001 for the calculations.

2.6 Techniques used for the training of ML models

We used 14 different ML algorithms available in the AutoGluon package (Erickson et al., 2020) for the training and performance evaluation of ML models (Table 2).

2.7 PSI-BLAST-based classification of proteins

We used two approaches to evaluate PSI-BLAST's performance for classifying virulent and nonvirulent proteins. Firstly, a local database of training dataset protein sequences (n = 5425) was generated, and a sequence similarity search of test dataset sequences (n = 1356), against this database was performed through PSI-BLAST (using the same PSI-BLAST run parameters as mentioned in the Section 2.3). Secondly, the PSI-BLAST search performance (using the same PSI-BLAST run parameters as mentioned in the Section 2.3) was also evaluated through a five-fold cross-validation technique applied to the training datasets protein sequences. For a five-fold cross-validation-based PSI-BLAST evaluation, the virulent and nonvirulent protein sequences training dataset was shuffled and divided into five equally sized parts. The PSI-BLAST search was performed five times so that each time, one part was used as a test set, and the remaining four parts were used to build a local database against which the search was performed. Thus, each part was used at least once for training and testing.

2.8 Criteria used for the selection of best ML models

The training dataset is used for training the machine learning models (0.9 fraction) and identifying the best-performing models with the help of the validation dataset (holdout 0.1 fraction from the training dataset). The models with the highest accuracy on the validation dataset are automatically opted for and saved as the best models by AutoGluon (Erickson et al., 2020). The saved models are further evaluated with a test dataset to estimate their real-life performance. The best-performing models on validation and test datasets are deployed on the VirulentPred 2.0 web server. Moreover, the best model (PSSM-based) is also available on the VirulentPred 2.0 web server for their usage as online predictors by users on their desktops or workstations.

3.1 Whole AAC-based models

Firstly, the best model trained with the whole AAC of protein sequences achieved the highest accuracy of 80.85% and 74.85% for validation and test datasets, respectively (Table 3). For DPC-based models, the highest accuracy of 79.37% and 74.56% was achieved for validation and test datasets, respectively. Similarly, 74.40% and 69.91% accuracy were achieved for TPC-based models for validation and test datasets, respectively. Thus, all the AAC-based models performed with moderate accuracy.

3.2 N and C-terminus residues' AAC-based models

The AAC of N and C-terminus protein/peptide sequences are essential features in predicting various biological properties, for example, antimicrobial, antibiofilm, cell-penetrating, tumor homing, and various other activities (Cooper & Marsden, 2017; Gautam et al., 2013; Krishna & Englander, 2005; Lata et al., 2010; Petsalaki et al., 2006; Sharma et al., 2013, 2016; Yamada et al., 2017). However, in the present study, the highest accuracy of 59.51% (with test dataset) was achieved with the AAC of the C-terminal first 10 residues (CT10) as input features to the machine learning models (Table 4).

3.3 N and C-terminus residues' binary profile pattern-based models

Specific residues at the N and/or C-terminus of the sequences (in a position-specific manner) have been reported to be associated with the biological activities of proteins and peptides (Sharma et al., 2013, 2016). Therefore, in the present study, an attempt has been made to predict the virulent protein sequences with the help of N and C-terminus residues' BPPs. As evident from Table 5, the highest accuracy of 57.51% (test dataset) was achieved with the BPPs of the N-terminal first 10 residues (NT10) as input features to the machine learning models.

3.4 Performance of hybrid models

Eleven hybrid ML models were developed with all possible combinations of the calculated input features (Table 6). The highest test dataset accuracy of 84.51% was achieved with a hybrid of AAC, DPC, and PSSM as input features to train the ML model.

3.5 PSI-BLAST-based classification of virulent and nonvirulent proteins

Retrospectively, while working on the VirulentPred server (Garg & Gupta, 2008), we have seen that PSI-BLAST (Altschul et al., 1997) alone cannot efficiently classify virulent and nonvirulent protein sequences. Using PSI-BLAST as a similarity-based search model, we could predict 52.5% and 51.7% of virulent and nonvirulent proteins, respectively, leading to an overall accuracy of 52.1%. Therefore, AI methods always have the edge over conventional similarity-based search methods.

A two-step process was used to check the same for the present study. A PSI-BLAST-based search was performed with test dataset proteins in the first step. Thus, search hits were retrieved for only 71.46% of test dataset protein sequences (n = 969) when no identity (%) threshold was applied (Table S3). During the search, if a test dataset protein sequence showed identity with more than one training dataset of protein sequences, the functional class of the highest identical subject protein (with the highest identity % value) was assigned as the predicted functional class (virulent or nonvirulent) to the query test protein. Thus, PSI-BLAST failed to detect hits for 28.54% of test dataset proteins (n = 387) at no identity (%) threshold. Although higher accuracy was achieved with BLAST, but with a fewer protein sequences than the ML-based method. However, for a rational sequence identity/similarity-based functional class assignment, an identity threshold of at least ≥30% is used. At this identity threshold of ≥30% between query and subject proteins, BLAST could annotate just 55.53% of test dataset proteins. Thus, it is clear from this analysis that BLAST failed to detect 44.47% of test dataset protein sequences even when evaluated with a small and well-curated dataset of annotated virulent and nonvirulent proteins. Moreover, at the highest identity thresholds of ≥50, hits were retrieved for only 6.05% of test dataset protein sequences.

In the second step, only 68.85% of protein sequences retrieved hits showed similarity with the local sequences database through a five-fold cross-validation technique when no identity cut-off was applied (Table S4). Functional classes were assigned to query test dataset sequences following the rules used in the first step. At an identity cut-off of ≥30%, only 52.70% of protein sequences retrieved hits. At the highest identity thresholds of ≥50%, the hits were retrieved for only 5.33% of protein sequences. Thus, it is clear from these two evaluations that PSI-BLAST is not sufficient to identify the virulent proteins.

3.6 Comparative performance of VirulentPred models

The comparative evaluation with the latest test dataset (from VirulentPred 2.0) helped assess the performance improvement achieved by VirulentPred 2.0 over the previously developed VirulentPred. Table 7 provides the comparative performance of the best models from VirulentPred 2.0 and VirulentPred 1.0. It can be seen that VirulentPred 2.0 is more accurate than its previous version. For example, in the case of the “Cascade SVM classifier”, an approximately 7% increase in the prediction accuracy (from 75.74% to 82.82%) is achieved with VirulentPred 2.0. Whereas, for PSSM profile-based models, about an 11% increase in the prediction accuracy (from 74.19% to 85.18%) is achieved with VirulentPred 2.0. In the case of threshold independent performance evaluation, a significant improvement in the AUC value, that is, from 0.85 (reported in the previous version of VirulentPred, for the PSSM-profiles-based model) to 0.924 is achieved (Figure S1). Therefore, the PSSM profile-based model is deployed as the best predictor model in VirulentPred 2.0.

3.7 Advantages of VirulentPred 2.0 standalone over web-server

VirulentPred 2.0 web-server version can be used by biologists who wish to make predictions for a few sequences or those who do not have any knowledge of software installation. The standalone version can be installed on local desktops/workstations for bulk classifications. Furthermore, depending on the availability of hardware configuration (at the user's end), multiple runs can be applied simultaneously by supplying multiple input sequences files (after splitting thousands or lakhs of sequences containing big files into smaller hundreds of sequences containing files). Moreover, long waiting queues on the web server can be avoided using the standalone VirulentPred 2.0.