Adaptive Residue-wise Profile Fusion for Low Homologous Protein Secondary Structure Prediction Using External Knowledge

1 Introduction

The structure of a protein is closely related to its function. Therefore, accurate protein structure estimation is of great importance, especially for drug development (e.g., vaccine development). X-ray crystallography, nuclear magnetic resonance(NMR) [Wuthrich, 1989] and cryo-EM [Wang et al., 2015] are commonly used techniques to obtain the protein structure. However, these techniques have certain drawbacks: X-ray crystallography experiments are timeconsuming; NMR faces the maximum sequence length limitation; and cyro-EM's equipments are over costly. Therefore, the computer-assisted protein structure prediction technique is gradually attracting researchers' attention because of its superior accuracy and efficiency. This paper will use computational models to conduct the protein secondary structure prediction (PSSP). PSSP mainly deals with the local protein structures(i.e., coil, alpha-helix, beta-sheet) rather than the overall structure (i.e., tertiary structure) because directly predicting the overall structure is hard to optimize and generally results in poor performance. Therefore, accurate prediction of each residue's local structure will be helpful for assembling the tertiary structure of a protein. However, using pure protein sequence as input to predict secondary structure is still challenging. An alternative approach that could improve the accuracy of PSSP is to provide protein homologous information e.g., multiple sequence alignment (MSA), as the additional input features.

∗Corresponding author. † Equal first authorship.

For the existing PSSP models [Li and Yu, 2016; Wang et al., 2016; Zhou and Troyanskaya, 2014], Positional-Specific Scoring Matrix (PSSM) is a widely used input feature calculated from MSA, whose effectiveness has been demonstrated experimentally. Specifically, [Zhou and Troyanskaya, 2014] utilized PSSM as input features and adopted the convolution neural networks (CNNs) to deal with PSSP. [Wang et al., 2016] further applied the conditional random field after CNN layers to model the relationship between input elements. [Guo et al., 2020] combined LSTM and CNN to extract both global and local features in parallel to obtain more fine-grained representations. All these methods have achieved remarkable performance under the guidance of PSSM. However, for low homologous proteins, it cannot find sufficient MSA to obtain a highly representative PSSM, which causes above models to suffer from obvious performance degradation. As shown in Fig. 1, the model performance for low homologous proteins(i.e. MSA count <10) is at least 10% lower than that of high homologous proteins(i.e. MSA count>2000). To improve PSSM quality, [Guo et al., 2020] proposed a self-supervised model to learn the mapping from the lowquality PSSM to the high-quality ones, similar to image super-resolution. But it still performs poorly (around 60% accuracy) on low homologous proteins (i.e.MSA count<60). Because it only focuses on minimizing the difference between enhanced PSSM and high-quality PSSM straightforwardly without jointly optimizing them on semantic level for PSSP.

The profile (also calculated from MSA) is an alternative solution. Previous works utilize PSSM instead of profile as input features because PSSM usually performs better than the profile for PSSP. However, in this paper, we demonstrate the superiority of profile over PSSM for low homologous proteins. Specifically, we firstly analyze the reason why profile works better than PSSM on low homologous cases and then experimentally verified the high performance of profile for the proteins with low MSA count. As shown in Fig. 1, profile outperforms PSSM by 2% for low homologous proteins on publicly available BC40 dataset. Based on this observation, we further introduced the self-supervised BERT's output [Rives et al., 2019] as the pseudo profile, which is adopted as the external knowledge to supplement the original low-quality profile. Furthermore, we designed another adaptive residue-wise (i.e., token wise) profile fusion to fuse both the BERT profile and the low-quality profile to get the enhanced profile. Unlike previous self-learning attention, the proposed attention is explicitly supervised by pseudo labels which guide the model to focus more on the profile matrix's valuable column features. Considering alignment from enhanced PSSM to high-quality one is essential for accurate PSSP, simple alignment with MSE loss [Guo et al., 2020] on PSSM itself ignores the feature difference on high-level feature space (e.g., in PSSP). Thus, we propose the feature consistency loss as well, which constrains the consistency of different features frommultiple semantic spaces, as shown in Fig. 2. Our contributions are three folds:

• We analyzed and demonstrated the superiority of profile over PSSM for low homologous proteins. Based on the observation, we shifted the PSSM enhancement to profile enhancement for low homologous proteins.
• We introduced the BERT pseudo profile as the extra knowledge to complement the low-quality profile. A novel residue-wise profile fusion with supervised attention loss was designed to combine two profiles (i.e., BERT pseudo profile and low-quality profile) in a finegrained manner. Besides, a feature consistency loss was specially constructed to align the enhanced profile to the high-quality one in multiple semantic levels.
• Extensive experiments on the three public datasets showed the superior performance of the proposed model (i.e., 4.7% improvement against previous state-of-theart method [Guo et al., 2020] and 7.3% improvement against low-quality profile for extremely low homologous proteins).

2 RelatedWorks

2.1 Multiple Sequence Alignment (MSA)

MSA aligns a target protein sequence with multiple homologous protein sequences [Wang and Jiang, 1994], which is a key technique for modeling sequence relationships in computational biology. For a protein sequence, MSA is searched by conducting pairwise comparisons [Altschul et al., 1990], Hidden Markov Model-like probabilistic models [Eddy, 1998],[Johnson et al., 2010], [Remmert et al., 2012], or a combination of both [Altschul et al., 1997] to align the sequence against a given protein database. Based on the MSA, researchers usually calculate the Position-Specific Scoring Matrix (PSSM) as features used for subsequent tasks.

2.2 Low-quality PSSMEnhancement

MSA and PSSM are critical information for protein property prediction. "Bagging" [Guo et al., 2020] is the first attempt to enhance the low-quality PSSM. By minimizing the MSE loss between the reconstructed and the original high-quality one, "Bagging" uses self-supervised method to reconstruct highquality PSSM from low-quality one generated by downsampling. Although "Bagging" has achieved relatively satisfactory performance, there are still some limitations. Firstly, it uses a fixed ratio for MSA downsampling to obtain low quality PSSM, which reduces the robustness of the "Bagging" model, especially for sequences with very few homologous proteins. Secondly, "Bagging" only enhances PSSM and ignores the joint optimization of PSSM and downstream PSSP.

2.3 Self-supervised Protein Sequence Modeling

Self-supervised representation learning is a powerful tool to learn from unlabeled data [Peters et al., 2018; Yang et al., 2019]. Similar to natural language, unlabeled protein sequences also contain important biological knowledge. Recently, protein sequence representation learning has demonstrated positive results on many downstream tasks including secondary structure prediction [Alley et al., 2019; Bepler and Berger, 2019; Rao et al., 2019; Rives et al., 2019]. TAPE [Rao et al., 2019] is the first to systematically evaluate the protein sequence modeling. They assessed the performance of three common types of pre-trained protein models,i.e.recurrent, convolutional, transformer models. They also proposed a benchmark dataset for five downstream tasks including secondary structure prediction. We chose the transformer based BERT model for its better generalization reported in the TAPE. After pre-training, the BERT model is able to provide additional information like embedding features for protein with zero homology.

2.4 AttentionMechanism

Attention is widely used in deep learning which could help model focus more on interesting parts. In [Vaswani et al.,

2017], the authors applied attention mechanism in neural machine translation and achieved high accuracy of translation. Later on, attention was introduced into various computer vision tasks. [Hu et al., 2018] designed the channel attention for image classification to filter out redundant channels. [Woo et al., 2018] further proposed the spatial attention to pick out the discriminative regions. To capture the long-distance relation and enlarge the receptive field, self-attention is adopted in [Wang et al., 2018], which calculates the similarity between any two points. All above attention mechanisms are learned implicitly. In this paper, we explicitly guide the learning of the attention with the supervision of pseudo labels, which helps avoid the wrong focus. After training, we use this attention to residue-wisely fuse different input features and get more robust representations.

3 Method

In this section, we propose a new framework for low homologous PSSP by profile enhancement. Specifically, we firstly demonstrate that profile is more appropriate than PSSM on this task. Besides, the residue-wise profile fusion is designed to adaptively aggregate different profiles (i.e.low-quality profile, pseudo profile from external pre-trained protein BERT model). Finally, a feature consistency loss is exploited to align the enhanced profile to high quality one from multiple semantic levels.

3.1 Comparison between Profile and PSSM

Frequency matrix F is adopted both in PSSM and profile, which counts the frequency of an amino acid position-byposition from N homologous proteins (i.e., the MSA count is N). Based on F, both the profile and PSSM could be derived, as shown in Eq. 1.

where θ is the pseudo count and takes 1 in practice. B is the prior background frequency which is a constant vector. Because of F < N, profile is obviously a normalized matrix whose element value p ∈ [0, 1]. However, PSSM values vary greatly due to the log operation. Especially when MSA count N → 0 (i.e.low homologous proteins), PSSM corresponds to large negative values, which will make the training process of existing PSSP models unstable. Besides, to quantitative analyze the the effect of PSSM and profile, we have conducted multidimensional experiments under different numbers of homologous proteins, as shown in Fig. 1. For low homologous proteins, profile outperforms PSSM by 2%. Therefore, profile is a better choice for low homologous PSSP.

3.2 Pseudo BERT Profile Generation

[Rives et al., 2019] exploits a large scale protein database (i.e., 250 million) to model co-evolution of proteins by using a BERT model in a self-training manner. Inspired by it, we introduce the output of the pretrained BERT into our model as the external knowledge to improve the low quality profile enhancement. More specifically, given a protein sequence S with length L, in each iteration, we mask a specific token t in S and utilize other tokens to predict the amino acid type for current token t using a pretrained BERT. Here, we extract the probability vectors after softmax as the one column in the profile matrix for residue t. Hence, after L iterations, a profile matrix Pb with shape L × 20 for protein S can be achieved. Finally, BERT pseudo profile Pb will be utilized as additional knowledge for low quality profile enhancement in our model by residue-wise profile fusion.

3.3 Adaptive Residue-wise Profile Fusion

To aggregate the low-quality profile Pl with the BERT pseudo profile Pb, we adopt a fine-grained residue-wise fusion approach which can adaptively fuse two profiles by assigning each residue column a specific weight for Pl and Pb. As shown in Fig. 2, we firstly train two PSSP Net F 1 p , F 2 p for 3state PSSP, where F 1 p is only trained by low quality profile Pl and F 2 p is only trained by pseudo profile Pb. Then, we exploit two PSSP Net outputs F 1 p (Pl), F 2 p (Pb) (i.e.probability vectors with shape L × 3 for each) to predict the weight vectors u, v by grading net Fg, since the probability vectors contain very representative confidence information for 3-state PSSP which reflects the profile quality for each residue column in matrix Pl, Pb.

Different from previous attention mechanism, we design a novel supervised attention loss to guide the model to focus more on discriminative residue columns in profile matrixes and reduce the difficulty of optimization. More specifically, the grading net Fg and predicted weight vectors u, v are supervised by a residue-wise attention loss Lr, as shown in Eq. 2

where u′ , v′ are the pseudo labels which are generated from PSSP errors of pretrained PSSP Net F 1 p , F 2 p . Specifically, given PSSP label Y and predictions F 1 p (Pl), F 2 p (Pb) , we construct pseudo label u′ , v′ in Eq. 3 where CEi is the crossentropy error for i-th residue between prediction and PSSP label Y . We use u′ , v′ to represent the confidence of the model for different features. During training, we use pseudo labels u′ , v′ to supervise the model to learn the adaptive attention, which could avoid the model to focus on the wrong features.

Finally, the predicted weights (i.e.u,v) fuse two profiles Pl, Pb as uT Pl +vT Pb and input to enhance net Fe for profile enhancement.

3.4 Feature Consistency

To reduce the gap between the enhanced profile Pe and the high quality profile Ph, we introduce the feature consistency loss Lf to constrain the similarity of features from multiple semantic levels. Different fromMSE loss [Guo et al., 2020] which focuses on the PSSM itself, Lf could optimize enhanced network Fe through a PSSP task. Specially, we first pretrain a two-layer stacked BiLSTMs as the feature extractor by optimizing a PSSP task. After that, we fix the parameters of feature extractorFx and collect features frommultiple layers of extractor. Then, we use Lf to minimize distance of corresponding semantic features between enhanced profile Pe and high-quality profile Ph.

Eq. 4 shows the whole process, where F 1 x is the first layer of extractor(i.e., the first BiLSTM) and F 2 x is the softmax output after fully connected layer. Since the extractor is optimized for PSSP. Therefore, a cross entropy loss is utilized to minimize the classification error with label Y .

Finally, Lf could supervise the features learning where KL is the Kullback-Leibler divergence.

3.5 Loss Function

As shown in Fig. 2, our model is supervised by three losses (i.e., residue-wise attention loss Lr, feature consistency loss Lf and MSE loss Lm). Similar to "Bagging" [Guo et al., 2020], we also introduce MSE loss to minimize the gap between the enhanced profile Pe with its ground truth Ph. Since the range of values in profile matrix is [0, 1] which causes very small gradients by MSE, to tackle this issue, we do some transformation on Ph by applying a logarithm operation as shown in Eq. 5 where ρ takes 0.001 in practice to avoid 0

value in profile.

These losses are optimized jointly as L = Lr + Lf + Lm with equal trade-off weights.

4 Experiments

Extensive experiments on three public-available datasets have been conducted to prove the effectiveness of our approach.

4.1 Dataset

Three public-available datasets (i.e., CullPDB, BC40 and CB513) is utilized to optimize and evaluate the proposed approach. The training set of CullPDB is utilized as our training set and we examine the performance on validation set of CullPDB, BC40 and CB513. The details of three datasets are shown in Tab. 1. CullPDB dataset [Wang and Dunbrack Jr, 2003] consists of 6125 protein sequences. There is less than 25% protein identity between any two sequence in the dataset. We take the same train-valid spit strategy as [Zhou and Troyanskaya, 2014] which adopt 5600 proteins for training and 525 for validation. BC40 is a large scale dataset constructed by 36976 proteins which are selected from PDB database by 40% sequence identity cutoff. Besides, the dataset also make sure there is no proteins share more than 20% sequence identity with CullPDB dataset. CB513 dataset is proposed in [Avdagic et al., 2009] and we follow [Kryshtafovych et al.,

2014] to remove redundancy. The multiple sequence alignments (MSA) are searched on Uniref90 database [Suzek et al., 2015]. The 3-state PSSP labels are generated by DSSP in [Kabsch and Sander, 1983].

4.2 Implementation Details

Pytorch is utilized to implement our work. One NVIDIA Tesla A100 GPU with memory 40 GB is exploited for the training, evaluation and BERT pseudo profile generation. We load the weights of BERT from released pretrain model 1 by [Rives et al., 2019]. Since the maximum positional embedding of this model is 1024, so we exclude the sequences with length large than 1024 in the training and validation. All loss functions are optimized jointly by Adam optimizer with learning rate 0.001. The network is trained in an end-to-end manner and dropout ratio is 0.5 for each BiLSTM. The number of training epochs is 100. All details will be included in SourceCode in supplementary files.

4.3 Network Architecture

All PSSP Nets F 1 p , F 2 p and Fx share a same architecture which consists of two-layer stacked BiLSTM and two FC layers behind for 3-state PSSP classification. In practice, we utilize F 2 p as the extracor Fx , which shares weights between two networks. As shown in Fig. 2, we exploit grading net Fg to predict the weight for each residue which adopts two stacked BiLSTM to extract features from the two probability maps F 1 p (Pl), F 2 p (Pb) with shape L × 3 for each. Then, one FC layer is followed to generate weight vectors u, v with shape L × 1 for each. The architecture of enhance net Fe is similar with [Guo et al., 2020] that consists of two paths which are 1-D CNN and BiLSTM to learn local and global features concurrently. Finally, two FC layers are used to combine two path features and yield final enhanced profile.

4.4 Result

We conduct comparison experiment and ablation study to compare with previous state-of-the-art method "Bagging" [Guo et al., 2020] and finely examine the gains of each component in our approach. Besides, a qualitative visualization will further illustrate the superior performance of proposed approach. Top-1 per protein accuracy is calculated for statistics the accuracy of the model. We evaluate the proposed approach on different magnitude low homologous proteins which can be partitioned by MSA count and Meff score. More specifically, we do comparison of PSSP accuracy with different MSA count (i.e., < 10,< 30,< 60) and various Meff scores (i.e., < 5,< 15,< 25,< 35). A more detailed comparison on specific MSA counts is shown in Fig. 3

Comparison Experiment. We compare the PSSP accuracy of our enhanced profile with "Bagging" [Guo et al., 2020] on three public available datasets with different MSA counts in Tab. ??. By comparison, we observe that the PSSP accuracy of our approach surpasses previous state-of-the-art method "Bagging" with a large margin (i.e., 4.7% on extremely low homologous cases) on all evaluated datasets.

1 https://github.com/facebookresearch/esm

Comparing with the unrefined low-quality profile, our enhanced profile gains a 7.3% improvement on PSSP accuracy. Particularly, since there is too few low homologous proteins in CB513 dataset, we only provide the MSA count partition ≤ 60 in Tab. ??.

Since MSA count score cannot precisely reflect the homology of proteins (i.e., MSA count is large but high redundancy), we further do comparison between our approach on different Meff score partitions as shown in Tab 3. Particularly, Meff score represents the number of non-redundant sequence homologous which can be calculated by Eq. 6 where Ti,j is a binarized similarity score of two proteins.

Ablation Study. Extensive ablation studies are conducted on BC40 dataset to evaluate the accuracy improvement of each component in our approach. We first remove the profile fusion module and only utilize low-quality profile as input for profile enhancement. As shown in Tab. 4, comparison with 'w/o Fusion' illustrates PSSP accuracy reduces dramatically, especially for extremely low quality profile (i.e., from 0.755 down to 0.717 proteins with MSA count 0), which exactly proves the effectiveness of proposed profile fusion approach and successfully introduce the external prior knowledge from BERT pseudo profile. To examine the gains from proposed supervised attention loss Lr which is utilized to supervise weight vectors u, v, we remove this loss to compare with the full model which is shown as 'w/o SA' in Tab. 4. Obviously, without supervised attention loss Lr, the grading net Fg can not provide precise weights to enhance residues with representative column in profile matrix(i.e., degradation from 0.787 to 0.773 for partition ≤ 30). We notice than no reduction for 'w/o SA' on partition = 0, since for proteins with MSA count 0, their low-quality profiles are the matrixes with all values equal to 1 which is useless at all, hence, supervised attention loss cannot provide gains on this special case. Moreover, we also specify the improvement of proposed feature consistency loss Lf by removing the this supervision during the training phrase as shown 'w/o FC' in Tab. 4. All partitions accuracy are reduced with large margins (e.g., 2.6% degradation for ≤ 30), which strongly proves the effectiveness of feature consistency loss and shows the importance of semantic feature alignment for profile enhancement.

Qualitative Visualization. To do detailed comparison, we further provide qualitative visualization for a specific low homologous protein as shown in Fig. 4 which compares the quality of MSA sampled from low-quality profile and our enhanced profile. We can clearly notice that the enhanced profile is far representative than low-quality profile which exactly proves the superior performance of our method and achieves state-of-the-art performance in visualization qualitatively.

5 Conclusion

In this paper, we focus on the secondary structure prediction for low homologous proteins. Experimental and theoretical justification show that profile performs better than PSSM on these proteins with few MSAs. Inspired by this, we further introduce BERT output as the external pseudo profile to complement the original low-quality one. Specifically, an adaptive residue-wise profile fusion is designed with a novel supervised attention loss to fuse different profiles. During fusion, strengths of different features will be selected out to form more robust representations. Besides, to align different profiles from multiple semantic levels, we introduce the feature consistency loss. Through the novel design, we improve the performance for low homologous proteins second structure prediction significantly.

Acknowledgements

The work was supported in part by Key Area R&D Program of Guangdong Province with grant No. 2018B030338001, by the National Key R&D Program of China with grant No. 2018YFB1800800, by Shenzhen Outstanding Talents Training Fund, by Guangdong Research Project No. 2017ZT07X152, by NSFC-Youth 61902335, by Guangdong Regional Joint Fund-Key Projects 2019B1515120039, by helix0n biotechnology company Fund and CCF-Tencent Open Fund.

2 https://weblogo.berkeley.edu