CSGNN: Contrastive Self-Supervised Graph Neural Network for Molecular Interaction Prediction

1 Introduction

There are various molecular interactions/associations in biological systems, such as drug-target interactions [Cheng et al., 2012], drug-drug interactions [Zhang et al., 2017], proteinprotein interactions [Kov´acs et al., 2019], drug-disease associations [Yu et al., 2020], and miRNA-disease associations [Huang et al., 2020a], and they implicate significant information for sophisticated biological system analysis [Huang et al., 2018]. Although many interactions have been revealed by the wet experiments or clinic reports, a portion of interactions remains undiscovered. However, wet experimental methods for identifying molecular interactions are extremely costly and time-consuming. The computational methods, in contrast, can efficiently discover potential interactions and guide in vivo validation, which provides an economic alternative.

∗Contact Author

Since molecular interaction discovery is a prerequisite for the downstream analysis (e.g., disease diagnosis and drug discovery), a great number of computational methods have been proposed to reveal molecular interactions in one specific domain (e.g., drug-drug interactions and drug-target interactions). Existing methods heavily depend on features and structures involving the molecules in one specific task, which limits the generalization capacity of transferring the models to other tasks. Thus, the generalized models for multifarious molecular interaction prediction are in demand.

Molecular interactions can be naturally formulated as networks where molecules and interactions are regarded as nodes and edges respectively. Then, network-based algorithms such as random walk and label propagation can be applied for molecular interaction prediction (MIP). Recently, network/graph-based deep learning techniques, e.g., graph neural networks (GNNs) [Wu et al., 2021], have emerged as a dominant class of methods which advance network analysis. In the light of GNNs, many tasks, such as node classification, graph classification, and link prediction, have been successfully handled. In GNNs, one of the most popular categories is graph convolutional networks (GCNs) [Kipf and Welling, 2017] which learn node representation by neighborhood aggregation and feature transformation. GCN provides a flexible and powerful frame for representation learning, which motivates us to resolve molecular interaction prediction problems.

A desired molecular interaction prediction method should have transferable, generalized, and robust performances, whereas several bottlenecks have restricted it. On the one hand, in molecular interaction networks, the degree distribution of nodes links are often nonuniform and sparse [Liu et al., 2020], which challenges many methods. Additionally, in many GNN frameworks, due to the immediate neighborhood aggregator, node representations learned by stacking multiple layers are over smoothed and redundant, which undermines the effect of the indirect nodes. On the other hand, the scarcity of annotated data restricts the predictive capacity of models. Typically, we can only acquire the few molecular interactions and features involving the entities, but these specific features are usually not transferable. Therefore, the generalized methods are expected to make prediction merely based on limited interactions.

In this paper, we propose a contrastive self-supervised graph neural network (CSGNN) for molecular interaction prediction (MIP) while tackling the above challenges. CSGNN leverages a mix-hop [Abu-El-Haija et al., 2019] aggregator in the deep graph neural network framework aiming to model and capture the dependent relation between indirect neighbors. Further, a contrastive self-supervised graph neural network, which maximizes mutual information between local patches and the network-level context, is elaborated to assist CSGNN with effective representation learning from scarce data. The contributions can be summarized as:

• The proposed CSGNN can accurately predict interactions between multifarious molecular entities, merely based on known interactions, not depending on any features or structures of entities.
• In molecular interaction networks, we consider the importance of high-order dependency and design a deep mix-hop graph neural network to model the dependency, which is proved to be effective in ablation study.
• We introduce a contrastive self-supervised learning task as a regularizer of molecular interaction prediction within a multi-task learning paradigm, and develop a delicate framework CSGNN, which provides a novel insight for molecular representation learning.

2 RelatedWorks

In molecular interaction networks, the goal of molecular interaction prediction is to predict if a given pair of biomedical entities such as proteins, drugs, or diseases will interact. There are three types of network-based methods: (1) Similarity-based learning assumes that there likely exists a link if a pair of non-connected nodes are similar or close in the networks [Liu et al., 2020]. (2) Matrix completion transforms the interaction prediction as a matrix completion problem [Zhang et al., 2020b]. (3) Graph representation learning acquires representations of entities from networks for downstream link prediction [Zhang et al., 2020a]. Although these pioneers have produced good performances, a neglected deficiency is that most of them are designed for the specific molecular interaction tasks and cannot be transferred to other tasks.

Recently, the application of graph neural networks in multifarious molecular interaction prediction tasks has received increasing attention. For example, MR-GNN [Xu et al., 2019b] utilizes a dual graph neural network to predict structured entity interactions. SkipGNN [Huang et al., 2020b] infers interactions between molecules via a skip graph neural network. Generally, like most deep learning based methods, GNNs require massive amounts of labeled data for model training. But in practice, only few known interactions/associations in the molecular interaction networks are available. Therefore, how to learn powerful representations from networks with limited data is crucial for the development of high-accuracy predictive models.

A number of graph-specific self-supervised learning pretext tasks have been designed to learn powerful representations for nodes or graphs. Contrastive learning, an important category of self-supervised methods, which aims to discriminate relative instance pairs and negative-sampled pairs through implicitly serving the intra-pair proximity as supervisory signals, deserves much attention. For examples, a few classic graph embedding methods, such as LINE [Tang et al., 2015], struc2vec [Ribeiro et al., 2017] and GraphSage [Hamilton et al., 2017], define different kinds of node proximity; GCC [Qiu et al., 2020] aims to distinguish between similar subgraph pairs and other dissimilar ones; DGI [Velickovic et al., 2019] seeks to classify node-graph pairs and 'fake' counterparts; the method which Hassani and Ahmadi [2020] extend from DGI introduces graph data augmentation to expand contrastive samples; InfoGraph [Sun et al., 2020a] and GraphCL [You et al., 2020] use a similar contrastive learning framework to learn graph-level representations. Another well-designed self-supervised method is explicit in the paradigm of supervised settings (classification or regression) with the targets automatically generated from the data itself. For examples, M3S [Sun et al., 2020b] utilizes a clustering algorithm to yield pseudo labels in a selftraining framework for semi-supervised node classification; GPT-GNN [Hu et al., 2020] forces GNN to predict masked nodes and edges; S2GRL [Peng et al., 2020] takes the minimized hop counts between two nodes as the supervisory signals. In this paper, we extend DGI and formulate it as a selfsupervised auxiliary task to enhance the performance of link prediction task.

3 The Proposed CSGNN

3.1 ProblemFormulation

A molecular interaction network can be viewed as an undirected attributed graph G = (V, E,X), where V represents the set of nodes that correspond to molecular entities (e.g., drugs, diseases or proteins) and E ⊆ V × V denotes the set of edges indicating the existence of interaction between two entities in V. X ∈ R|V|×m is a feature matrix where each node in V is encoded as a predefined m-dimensionality attribute vector (e.g., a generated graph embedding or a one-hot coding). Specifically, the graph can be further represented as an adjacency matrix A ∈ {0, 1}|V|×|V| where Au,v = 1 if there exists a link from u to v in the graph (i.e., (u, v) ∈ E) and Au,v = 0 otherwise. Note that G is a undirected graph, hence, A is symmetric. Our goal of molecular interaction prediction is to learn a mapping function Θ(ω) : E → [0, 1] from edges to scores, where ω is parameter, such that we can obtain the probability that two arbitrary nodes interact each other [Huang et al., 2020b].

3.2 Overview

Figure 1 illustrates the overview of our proposed contrastive self-supervised graph neural network framework, which contains a deep mix-hop graph neural network undertaking the molecular interaction prediction task and a contrastive graph neural network implementing the self-supervised learning task. The deep mix-hop graph neural network captures the indirect interactions in the molecular interaction networks; the contrastive graph neural network makes use of the information derived from data itself to enhance the generalization ability of the predictor. Two tasks, incorporated into an objective function, are jointly trained.

In the following, we will describe technical details of the deep mix-hop graph neural network and the contrastive graph neural network.

3.3 DeepMix-Hop Graph Neural Network

Graph neural networks (GNNs) use the graph structure and node attribute to learn the high-level compact representations of nodes, denoted by H ∈ R|V|×d where hv = H[v, :] is the latent representation of the node v. In general, GNNs iteratively update the representation of a node by aggregating representations of its neighbors. After l iterations, the hidden state of node v can be denoted as: h(l) v = φ(l) (h(l-1) v , f (l) ({h(l-1) u : u ∈ N(v)})), where φ(l) (·) is a combination function (e.g., summation, element-wise product, concatenation operation or a neural network) of l-th iterations. N(v) denotes a set of nodes adjacent to v. Specifically, we initialize h(0) v =X[v, :].

To model the high-order proximity in molecular interaction networks, we design a deep mix-hop graph neural network, which can highlight the role played by indirect neighbors, as shown in Figure 1. The update rule of the hidden states of nodes in our model can be defined as:

where f (l) k (·) is an aggregation function of l-th iteration on k-th hop, and Nk(v) denotes the set of k-hop neighbors of v.

Later, we decode the representations of node u and node v into the probability of their interaction: ˆ puv = ρ (hu, hv), ρ(·) is a scoring function (e.g., inner product, bilinear function or a neural network). Then the loss of supervised molecular interaction prediction task can be formulated as:

where T is the sampled training set of node pairs and puv denotes true label for the interaction between nodes u and v.

3.4 Contrastive Graph Neural Network

The proposed CSGNN leverages a self-supervised learning task which conducts contrastive learning between the original networks and corrupted networks by maximizing mutual information.

Following DGI [Velickovic et al., 2019], the selfsupervised auxiliary task maximizes mutual information between a local patch (i.e., node-level representation) and its global context (i.e., graph-level representation). One original intention of above design is guiding the node representations to capture the global information of the entire graph. We employ a readout function, Γ : H ∈ R|V|×d → s ∈ Rd (i.e., s = Γ(H)), to summarize the obtained local patches into the global context s.

Then, we corrupt the original graph by an explicit (stochastic) corruption pipeline to convert the graph into a "fake" one:

Next, we utilize a noise-contrastive type objective with a standard binary cross-entropy (BCE) loss between the positive pairs (each pair consists of global information and one of local patches) and the negative ones (each pair consists of global information and one of corrupted patches) to model self-supervised leaning task. Naturally, we present the following loss function:

The contrastive learning mechanism maximizing mutual information between the original global context and local patches, to some extent, allows our model to be more "discriminative" for node representation learning. Further, we extend the contrastive learning mechanism from another view: maximizing mutual information between the corrupted global context and local patches, which can be viewed as the "dual problem" of above problem:

where ˜ s denotes the corrupted graph-level representation. This architecture, on the one hand, exploiting more information derived from data itself, enhances the generalization ability. On the other hand, it consider both views of the of network (i.e., original and corrupted views) simultaneously, which enables the model more "balanced" and robust.

Intuitively, to train a model by the missing link task and the self-supervised leaning task, we optimize the following objective that combines Eq.2, Eq.4 and Eq.5:

where α, β, γ are hyperparameters which balance the contribution of different tasks. The pseudo-codes of CSGNN are illustrated in Algorithm 1.

4 Experiment

4.1 Datasets and Settings

We consider seven publicly available network datasets. (1) ChG-Miner consists of 5,018 drugs which target 2,325 proteins via 15,139 drug-target interactions (DTIs). (2) ChChMiner contains 48,514 interactions between 1,514 drugs Algorithm 1 The Contrastive Self-Supervised Graph Neural Network Algorithm Input: Molecular interaction network G; functionf(·), φ(·), ρ(·), Π(·, ·), Γ(·), Ψ(·); parameters α, β, γ; training set T

Output: Θ(ω)

• 1: while CSGNN not converge do

2: H(0) =X 3: ( ˜X, ˜A) ←Π(X,A) 4: ˜H(0) = ˜X 5: for u ∈ V do 6: for l = 1, 2, ..., L do 7: for k = 1, 2, ..., K do 8: Calculate f (l) k ({h(l-1) v : v ∈ Nk(u)}) 9: end for 10: Update h(l) u via Eq.(1) 11: end for 12: end for 13: s ←Γ(H) 14: Calculate ˜H, ˜ s like line 5-13 15: for (u, v) ∈ T do 16: Calculate puv log ˆ puv + (1 - puv) log(1 - ˆ puv ) 17: end for 18: Calculate ℓ1 via Eq.(2) 19: for u ∈ V do 20: Calculate E(X,A)[logΨ(hu, s)] 21: Calculate E(X,A)[log(1 -Ψ(hu, ˜ s))] 22: end for 23: for v ∈ V ′ calculate loss like line 19-22 24: Calculate ℓ2 and ℓ3 via Eq. (4) and Eq. (5) 25: ∇ωαℓ1 +βℓ2 +γℓ3 26: end while 27: return Θ(ω)

(DDIs). (3) HuRI-PPI includes 23,322 interactions among 5,604 proteins (PPIs) in HI-III network. (4) DG-AssocMiner comprises 519 diseases which associate 7,294 genes through 2,1357 associations (DGIs). (5) DD-Miner is a dataset with 6,877 associations between 6,878 diseases (DIAs) (6) DCh-Miner contains 5,536 diseases associate 1,663 drugs via 466,657 associations (DDAs). (7) ChSe-Decagon includes 639 drugs, 10,184 side-effects and 17,499 associations (DSAs) among them. We download DTIs, DDIs, DGIs, DIAs, DDAs, and DSAs from BioSNAP [Marinka Zitnik and Leskovec, 2018], and collect PPIs from CCSB[Luck et al., 2020]. The detailed information is listed in Table 1.

Baselines. We compare CSGNN with several state-of-theart methods, which can be categorized as:

• Matrix Factorization (MF) decomposes the input data matrix to acquire low-dimension representations. Two representative MF-based methods are selected, including GraRep [Cao et al., 2015] and HOPE [Ou et al., 2016].
• RandomWalk (RW) utilizes random walk strategy on graphs to generate several node sequences. One most representative RW-based method Struc2Vec [Ribeiro et

mix-hop K = 2 and we use the random distribution ranging from 0 to 1 to initiate the feature matrix X on 128 dimensions.

We employ MLP with one hidden layer as the scoring function ρ(·). Concretely, the interaction prediction score can be defined as:

where ⊙ is element-wise product. Note that, in above equation, multiple operations are performed to model multifarious relationships between two entities.

In the contrastive graph neural network, empirically, we randomly corrupt permutation of initial feature matrix X to ˜X via corruption function Π(·, ·). Then, we select Mean as the readout function Γ(·), which is experimentally efficient for large-scale data. Therefore, the graph-level representation can be denoted by: s = Mean(H). Further, we instantiate the contrastive discriminator Ψ(·, ·) as σ (hT Ws). Here, we use sigmoid as the activation function to produce the score that represents probabilities of being a positive sample. Note that the graph neural networks which encode the original and corrupted network share the same parameters.

In the joint training, we set α = 1, β = 0.1 and γ = 0.1.

We run CSGNN and other compared methods on our workstation with 2 Intel(R) Xeon(R) Gold 6146 3.20GHZ CPUs, 128GB RAM, and 2 NVIDIA 1080 Ti GPUs. For more detailed parameter settings, please refer to the source code 1 .

4.2 Results and Analysis

In this section, we conduct experiments to compare CSGNN with all the baselines, where GCN and GIN are respectively used as GNN encoders for CSGNN. The AUC and AUPR scores of all models are shown in Table 2, and we have the following observations from the results. Firstly, CSGNN outperforms all the baselines and produces the most competitive performance on seven benchmark datasets. Considering MF-based methods, RW-based methods, NN-based methods,

1 A reference implementation of CSGNN may be found at https://github.com/BioMedicalBigDataMiningLab/CSGNN or https://github.com/ChengshuaiZhao0/CSGNN

al., 2017] is chosen as baseline.

• Neural Network (NN) captures hidden patterns underlying the network structure from input data to learn graph embeddings. Two most representative NN-based methods LINE [Tang et al., 2015] and SDNE [Wang et al., 2016] are chosen.
• Graph Neural Network (GNN) considers both topological information and node and/or edge information in graph. Several most representative GNN-based methods including GCN [Kipf and Welling, 2017], GAT [Velickovic et al., 2018], GraphSage [Hamilton et al., 2017], GIN [Xu et al., 2019a] and SkipGNN [Huang et al., 2020b] are selected.

Metrics. In all experiments, each dataset is split into training, validation, and test sets as the ratio 7:1:2. Then, we train models on the training sets, select models with the best performance on the validation sets, and evaluate them on the test sets. The area under the receiver operating characteristic curve (AUC) and the area under the Precision-Recall Curve (AUPR) are elected as two representative indicators. Each experiment is conducted ten runs with random splits of dataset and the average metric is computed to avoid bias.

Experimental settings. In the deep mix-hop graph neural network, we empoly GCN [Kipf and Welling, 2017] and GIN [Xu et al., 2019a] as message passing frameworks, and concatenate the representation from each hop, therefore, the node representation can be formulated by:

where || is a concatenation operation, σ is a non-linear activation function, ˆ dv = 1 + |Nk(v)|, w(l) k is a weight matrix, and ϵ(l) k is a learnable parameter. Note that GIN can model an injective function mapping initial matrix to node representations, which allows CSGNN to powerfully capture the distinct topological structure of the graph. In this paper, we set and GNN-based methods, CSGNN achieves the average improvement of 20.8%, 15.34%, 17.38%, and 11.68% in AUC values, and 35.56%, 26.59%, 37.63%, and 11.9% in AUPR values, respectively. The results demonstrate the superiority of CSGNN when performing on molecular interaction prediction. Secondly, CSGNN outperforms GCN and GIN by 2.58% and 1.7% in AUC values on average, and 2.54% and 1.85% in AUPR values on average, respectively. The results indicate that the consideration of the mix-hop context and the contrastive self-supervised learning strategy boosts the generalization ability of CSGNN, which will be further discussed in Ablation Study. Thirdly, CSGNN with different GNN encoders achieves different performances among benchmark datasets. For benchmark datasets with low density (<0.1%), CSGNN with GCN encoder (i.e., CSGNNGCN) performs better than CSGNN with GIN encoder (i.e., CSGNN-GIN), while for other benchmark datasets, CSGNNGIN performs better than CSGNN-GCN. At last, we notice that several baseline methods (such as GAT and LINE) work well in node/graph classification do not obtain good performances on molecular interaction prediction, because of the distinct characteristics of biological networks.

4.3 Ablation Study

To further investigate the importance of components: the mix-hop aggregator and the self-supervised learning task, we design the following variants of CSGNN:

• CSGNN without mix-hop (w/o MH) only aggregates immediate neighbors instead of mix-hop neighbors in CSGNN.
• CSGNN without self-supervised learning task (w/o

SSL) removes the self-supervised learning task, and constructs GNN model based on mix-hop.

• CSGNN without mix-hop and self-supervised learning task (w/o MH + SSL) has neither mix-hop nor the self-supervised learning task.
• CSGNN without molecular interaction prediction task (w/o MIP) is trained only based on the contrastive self-supervised learning objective. Then, the acquired node representations will be used in downstream link prediction task.

We conduct ablation study on seven datasets, and the results are shown in Figure 2. It is clear that CSGNN with the mix-hop aggregator and the self-supervised learning task achieves superior performances on most settings, and the removal of any will undermine the predictive capacity of CSGNN. Overall, the mix-hop is more efficient than the contrastive self-supervised learning task. Besides, by introducing the contrastive self-supervised learning task, the datasets with few annotations (e.g., DGI and DIA) benefit more significantly compared to the datasets with sufficient ones (e.g., DDA and DSA). It's worth mentioning that only depending on the contrastive self-supervised learning task, CSGNN (w/o MIP) still achieves amazing performances, especially on DSA and PPI datasets. In conclusion, both mix-hop and the self-supervised learning task are necessary for CSGNN.

4.4 Visualization Analysis

In this section, we explore node representations on the molecule level. We extract the learned representations of molecule pairs in the interaction networks from the trained CSGNN model, and project them into 2D space using tSNE [Maaten and Hinton, 2008]. Here, we take the ChChMiner(DDI) dataset as an example to compare CSGNN with GCN and SkipGNN. As illustrated in Figure 3, the proposed CSGNN can well distinguish the linked molecules (red) and unlinked molecules (blue); SkipGNN recognizes most of the non-interacting pairs; GCN differentiates part of molecular pairs. To further analyze the quality of learned representations, we calculate the inter-cluster and intra-cluster distances via silhouette scores. CSGNN, SkipGNN, and GCN have silhouette scores of 0.2792, 0.1777, and 0.1705 respectively, indicating that CSGNN allows more delicate representations for molecular interaction prediction.

5 Conclusion

In this paper, we focus on crucial but rarely studied multifarious molecular interaction prediction problem. Most existing methods are dedicated to specific molecular interaction prediction tasks, and cannot be well transferred to other tasks. the scarcity of annotated data restricts the predictive capacity of conventional GNNs. To address above limitations, we propose a novel method CSGNN which leverages the deep mix-hop graph neural network to capture high-order proximity and employs the contrastive graph neural network to enhance the transferable and generalized ability. We intuitively incorporate contrastive learning task into the self-supervised learning framework as auxiliary regularization. Comprehensive experiments on seven public datasets show that CSGNN outperforms classic and state-of-the-art molecular interaction prediction models. CSGNN can be applied to various novel molecular interaction prediction tasks merely based on annotated interactions. In the future, we will explore more powerful representation learning techniques for molecular interaction inference.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No.62072206, 61772381); Huazhong Agricultural University Scientific & Technological Selfinnovation Foundation; Huazhong Agricultural University Students Research Fund (SRF).

We thank anonymous reviewers for their insightful and helpful comments. Thanks for the discussions with the team member, Ph.D. candidate Haitao Fu, who provides a lot of insights for the work.