2.1 Datasets

Initially, all entries from humans, rodents, mammals, vertebrates, invertebrates, fungi and plants were extracted from UniProtKB release 2019_05⁸ resulting in 189 818 proteins from 7859 species. We further processed these sequences retaining only metazoa proteins with a UniProtKB subcellular location annotation supported by published experimental evidence (ECO code ECO:0000269) leaving 29 573 sequences. This set of proteins was further reduced by removing any protein that was less than 30 amino acids long or greater than 10 000 amino acids long. The set was split into two groups: the endomembrane system and secretory pathway; and “other.” 10 223 sequences remained in the endomembrane system and secretory pathway which were further split into membrane and non-membrane. The majority of sequences are labeled with more than one subcellular location, we found 5236 sequences that were labeled with only non-membrane locations, 4093 sequences that were labeled with only membrane locations, and 894 sequences that were labeled with both membrane and non-membrane locations, these were added to the membrane set.

The resulting set was then internally redundancy reduced using BLAST¹³ as follows: the first sequence in the set is kept; after N sequences from the set have been examined and K kept, sequence number N + 1 is aligned (using BLAST) against all K that have been kept, discarded if it is more than 80% identical to any of them, or added to them otherwise. The process carries on until all 10 223 sequences in the initial set have been considered. Finally, 7517 sequences were retained and no two sequences were more than 80% identical to each other. A subset of sequences that were added to UniprotKB after 2016 was set aside in the 80% Independent Test Set (ITS-80) to allow for objective benchmarking against other webservers that were developed after 2016. The remaining 6988 sequences after removal of the ITS-80 became the Train-80 and were divided into five parts. These five parts were used to form five folds that are used for 5-fold cross-validation. For each of the 5-fold subdivisions, one of the five parts was assigned as the test set, a second part was assigned as the validation set and the remaining three parts were combined to create the training set. The rest of the folds were formed by repeating the same procedure (See Figure 1).

In the resulting folds, the validation set was redundancy reduced with respect to the test set using BLAST with an e-value of 0.001. Then the test and validation sets were internally redundancy reduced using BLAST with an e-value of 0.001. Finally, the test set and validation sets were redundancy reduced with respect to their corresponding training set using BLAST with an e-value of 0.001.

Two subsets of ITS-80 were used to benchmark other available web servers. The ITS-80 set was internally redundancy reduced and redundancy reduced to less than 30% sequence identity with respect to Train-80 leaving 240 sequences. We call the resulting set the ITS. We then created a second “strict” set by internally redundancy reducing the ITS-80 set using BLAST with an e-value of 0.001, and then redundancy reducing it with respect to Train-80 using BLAST with an e-value of 0.001, leaving 118 sequences. We call this set ITS_strict. Table 1 shows the number of proteins belonging to each class in the Training, ITS and ITS_strict sets.

This rather complex strategy of redundancy reduction ensures that there is minimal similarity between the sequences in the test and validation set with respect to each other and to the training set (e-value ≤ 0.001). However, it allows for a larger training set with sequences with up to 80% sequence identity within the training dataset. Furthermore, the two independent test sets have been redundancy reduced with respect to the training set to ≤ 30% sequence identity or e-value ≤ 0.001 ensuring that these two sets are indeed independent of the training set. Although, reducing redundancy on the entire set of proteins at the beginning of the procedure would be more straightforward, this would have resulted in a much smaller training set.

2.2 Alignments and data encoding

We generated alignments of multiple homologous sequences (MSA) for all datasets used in SCLpred-MEM by iterating PSI-BLAST¹³ for three rounds with an e-value of 0.001 against the June 2016 version of UniRef90.⁸ These alignments were encoded into MSA profiles by calculating frequencies of residues and gaps. The frequencies of the amino acids present in the original sequence were then “clipped” to 1, similar to^{2, 4} where each amino acid was represented by a vector of 22 number frequencies of an amino acid type from the list of homologous sequences.

2.3 Predictive architecture

In this work, we tested configurations of N-to-1 neural networks of various depths and width during the preliminary experiments. N-to-1 nets are composed of an input kernel mapping a window of amino acids into a feature vector followed by a stack of hidden convolutional kernels followed by an average pooling unit over the whole sequence, and a final fully connected network (Figure 2). The input Kernel, implemented as a two-layered feed-forward neural network, learns a nonlinear function I from a window of amino acids wc_i at position i and predicts an intermediate state vector vs_i at position i.

(1)

The hidden convolutional kernel, also implemented as a two-layered feed-forward neural network, learns a nonlinear function H^k at hidden layer k from a window of intermediate states at position i and predicts an intermediate state vector at position i.

(2)

The output vectors of the last hidden kernel at each position i are averaged element wise into a single vector v. A fully connected network predicts the final subcellular location cls of each protein from the final vector v. The fully connected network learns a nonlinear function O.

(3)

It should be noted that in our implementation, unlike standard Convolutional Neural Networks, all convolutional kernels and the final fully connected network are implemented by feed-forward neural networks with one hidden layer. That is, each convolutional kernel has two layers, and can be seen as a proper (1-layered) kernel followed by a nonlinearity, followed by a further kernel of size 1 and a further nonlinearity. In all cases, we use sigmoidal nonlinearities on the model's internal units, rather than rectified linear units. We found both the use of deeper convolutional stages and sigmoidal units to be beneficial in preliminary tests. The resulting architecture has a minimum of three internal (hidden) layers when no hidden-to-hidden convolutional kernel is present, while an architecture with k hidden-to-hidden kernels contains 3 + 2k hidden layers in total.

2.4 Hyperparameter tuning

An extensive hyperparameter search is essential to optimize the model for higher generalization capacity. We adopted a grid search approach for hyperparameter fine-tuning carried out on the first fold of the datasets. Input amino acid window sizes of 1 to 31 were tested and the generalization capability of the model deteriorated significantly after 31 amino acids. The models performed best with two hidden convolutional kernels each containing an input window size of 11 to 31.

The hyperparameters of the best performing model on the first fold of the validation set were considered as the optimal model. This optimal model consists of an input convolutional kernel followed by a stack of two hidden convolutional kernels followed by average pooling and two fully connected layers. This corresponds to a total of 7 inner layers containing roughly 540 weights with an effective input window of 21 amino acids.

2.5 Training and ensembling

The optimal model selected during the hyperparameter tuning phase was trained and tested in 5-fold cross-validation. In order to achieve a higher stability and greater generalization capability from the final system, the five best performing models were selected from each fold based on their performance on their respective validation sets. These best performing five models were ensembled within each fold to gauge the performance of each fold against the respective test sets. The final system contains an ensemble of 25 models selected across five folds on the basis of five best performing models per fold. This 25 model ensemble rectifies the performance fluctuation for unseen proteins and stabilizes the system for a better estimation of performance.

2.6 Performance evaluation

To evaluate the performance of SCLpred-MEM against other state-of-the-art predictors we measure specificity (Spec), sensitivity (Sens), Matthews correlation coefficient (MCC) and, accuracy (Acc)¹⁴ as follows:

(4)

where:

• True positives (TP): the number of sequences predicted in the membrane class that are observed in that class.

• False positives (FP): the number of sequences predicted in the membrane class that are not observed in that class.

• True negatives (TN): the number of sequences predicted not to be in the membrane class that are not observed in that class.

• False negatives (FN): the number of sequences predicted not to be in the membrane class that are observed in that class.

2.7 Benchmarking

The performance of SCLpred-MEM is compared to other state-of-the-art subcellular localization predictors. However, due to the high number of subcellular locations of eukaryotic cells, subcellular localization predictors classify query proteins into various locations within the cell. In order to compare them with SCLpred-MEM, we recast the predicted locations of other available predictors into membrane and non-membrane classes. Table 2 shows the mapping of predicted classes for the ITS and ITS_strict for each predictor.