Smart Contract Vulnerability Detection using Graph Neural Network

1 Introduction

Blockchain technology is developing rapidly due to its decentralization and tamper-free nature [Tsankov et al., 2018]. A blockchain is essentially a distributed and shared transaction ledger, maintained by all the miners in the blockchain network following a consensus protocol [Sankar et al., 2017]. Smart contracts are programs automatically running on the blockchain. However, ill-designed smart contracts expose vulnerabilities, which are perfect targets for network attacks. One notable example is the DAO event, where the hackers exploit the reentrancy bug of The DAO contract to steal 3.6 million Ether (Cryptocurrency of Ethereum). The case is not isolated and several security vulnerabilities are discovered and exploited every few months † . According to the statistics of SlowMist Hacked ‡ , blockchain networks have suffered more than 10 billion USD losses due to the security issues of smart contracts.

∗The first three authors are of equal contribution to this work. Zhenguang Liu contributes to the idea, Yuan Zhuang and Peng Qian contribute to implements and datasets. Zhenguang Liu is the corresponding author.

†The dao website, 2016. http://etherscan.io/address/0xbb9bc244 d798123fde783fcc1c72d3bb8c189413

‡Slowmist hacked website, 2019. https://hacked.slowmist.io/en/.

Current approaches for smart contract vulnerability detection are mainly inspired by existing testing methods from the programming language community, revolving around symbolic execution [Luu et al., 2016; Tsankov et al., 2018] and dynamic execution methods [Jiang et al., 2018; Liu et al., 2018b]. We scrutinized the released implementation of existing methods, and empirically observe that they suffer from two key problems. First, existing methods heavily rely on several expert-defined hard rules (or patterns) to detect smart contract vulnerability. However, expert rules are error-prone and some complex patterns are non-trivial to be covered. Crudely using several hard rules leads to high false-positive and false-negative rates, and crafty attackers may easily bypass the rules to perform attacks. Second, since the rules are contributed by a few 'centralized' experts who develop the detection tools, their scalability is inherently limited. As the number of smart contracts is increasing rapidly, it is impossible for a few experts to sift through all the contracts to design precise rules, while the knowledge of other 'decentralized' experts cannot be incorporated to improve the model.

Ourmethod. To address these problems, we propose novel methods beyond the rule-based framework. Specifically, we characterize the source code of a smart contract as a contract graph according to the data- and control- dependencies between program statements. Nodes in the graph represent critical function invocations or variables while edges capture their temporal execution traces. Since most GNNs are inherently flat during information propagation, we design an elimination phase to normalize the graph. We extend GCN to a degree-free GCN (DR-GCN) to handle the normalized graphs. Further, we take into account the distinct roles and temporal relationships of different program elements and propose a novel temporal message propagation network (TMP). We conducted extensive experiments on more than 300,000 real-world smart contract functions, results show that our approaches significantly and consistently outperform state-ofthe-art methods on the detection of different types of vulnerabilities including reentrancy, timestamp dependence, and infinite loop vulnerabilities. Our implementations are released to facilitate future research.

Contributions. To summarize, our key contributions are: i) We introduce a novel temporal message propagation network (TMP) and a degree-free GCN (DR-GCN) to automatically detect smart contract vulnerabilities. ii) We propose to characterize the contract function source code as contact graphs, and explicitly normalize the graph for highlighting the key nodes. iii) Our methods set the new state-of-the-art performance on smart contract vulnerability detection, and overall provide insights into the challenges and opportunities.

2 Problem Statement

Problem formulation. Presented with the source code of a smart contract, we are interested in developing a fully automated approach that can detect vulnerabilities at the function level. That is, we are to estimate the label ˆ y for each smart contract function SC, where ˆ y = 1 represents SC has a vulnerability of a certain type while ˆ y = 0 denotes SC is safe. In this paper, we focus on three types of vulnerabilities:

Reentrancy is a well-known vulnerability that caused the infamous DAO attack. In Ethereum, when a smart contract function F1 transfers money to a recipient contract C1, the fallback function of C1 will be automatically triggered. C1 may invoke back to F1 in its fallback function to reenter F1 for stealing money. Since the current execution of F1 waits for the transfer to finish, C1 can make use of the intermediate state F1 is in to succeed in stealing.

Infinite loop is a common vulnerability in smart contracts. The program of a function may contain an iteration or loop with no exit condition or the exit condition cannot be reached, i.e., an infinite loop. The fallback mechanism in smart contracts rises a new possibility of this non-termination bug, namely a cycled call between functions and the fallback function. For example, function A invokes function B with incorrect arguments, which will automatically trigger the execution of the fallback function in this contract. Suppose the fallback function further invokes function A, this will leads to a call loop between A and the fallback function.

Timestamp dependence vulnerability exists when a smart contract uses the block timestamp as a triggering condition to execute some critical operations, e.g., sending Ether or determining the winner of a game. The miner in Ethereum has the freedom to set the timestamp of a block within a short time interval (< 900 seconds) [Jiang et al., 2018]. Therefore, miners may manipulate the block timestamps to gain illegal benefits.

3 Our Method

Method overview. The overall architecture of our method consists of three phases: (1) a graph generation phase, which extracts the control flow and data flow semantics from the source code and explicitly models the fallback mechanism, (2) a graph normalization phase inspired by k-partite graph, and (3) novel message propagation networks for vulnerability modeling and detection. Next, we introduce the three phases, respectively.

3.1 Graph Generation

Existing work [Allamanis et al., 2018] has shown that programs can be transformed into symbolic graph representations, which are able to preserve semantic relationships between program elements. Inspired by this, we formulate a smart contract function into a contract graph, and assign distinct roles to different program elements (nodes). Further, we construct edges by taking their temporal order into consideration. Figs. 1(a) & (b) demonstrate a contract snippet and the graph constructed for its getBonus function, respectively.

Our first insight is that different program elements in a function are not of equal importance. Therefore, we extract three categories of nodes, i.e., major nodes, secondary nodes, and fallback nodes.

Major nodes construction. Major nodes symbolize the invocations to customized or built-in functions that are important for detecting the specific vulnerability. For example, for reentrancy vulnerability, a major node models the invocation to a transfer function or the built-in call.value function, which is key to detect reentrancy. For timestamp dependence vulnerability, the built-in function invocation block.timestamp is extracted as a major node. For infinite loop, all the customized functions within the contract are treated as major nodes. Formally, we characterize all the critical functions as major nodes, which are denoted by M1, M2, . . . , Mn.

Secondary nodes construction. While major nodes represent important invocations, secondary nodes are used to model critical variables, e.g., user balance and bonus flag. Formally, the critical variables are defined as secondary nodes S1, S2, . . . , Sn.

Fallback node construction. Further, we construct a fallback node F to stimulate the fallback function of an attack contract, which can interact with the function under test. The fallback function is a special design in smart contracts, and is the cause of many security vulnerabilities.

Edges construction. We further construct edges to model the relationships between nodes. Each edge describes a path that might be traversed through by the contract function under test, and the temporal number of the edge characterizes its order in the function. Specifically, the feature of an edge is extracted as a tuple (Vs, Ve, o, t), where Vs and Ve represent its starting and end nodes, o denotes its temporal order, and t the edge type. To capture rich semantic dependencies between nodes, we construct four types of edges, namely control flow, data flow, forward and fallback edges. The details of the semantic edges are listed in Table 1.

3.2 Contract Graph Normalization

Most graph neural networks are inherently flat when propagating information, ignoring that some nodes play more central roles than others. Moreover, different contract source code yield distinct graphs, hindering the training of graph neural networks. Therefore, we propose a node elimination process to normalize graphs.

Nodes elimination. As introduced in Section 3.1, the node of a graph is partitioned into major nodes {Mi}|M| i=1 , secondary nodes {Si}|M| i=1 , and the fallback node F. We remove each secondary node Si but pass the feature of Si to its nearest major node. Note that if Si has multiple nearest major nodes, its feature is passed to all of them. The fallback node is also removed similar to the secondary node. The edges connecting to the removed node are preserved but with their starting or end node moving to the corresponding major node.

Feature of major nodes. Features of major nodes are updated by aggregating features from their neighboring removed nodes. To distinguish between the original major node and its corresponding major node after aggregation, we denote the new major node of Mi as Vi. The feature of Vi is composed of three parts: i) self-feature, namely the feature of major node Mi; ii) in-features, namely features of the secondary nodes {Pj}|P| j=1 that are merged to Mi and having a path pointing fromPj toMi; and iii) out-feature, namely features of the secondary nodes {Qk}|Q| k=1 that are merged toMi and having a path directs from Qk fromMi. Fig. 1(c) illustrates the normalized graph of Fig. 1(b).

3.3 Message Propagation Neural Networks

In this subsection, we first extend the GCN to a degree-free GCN (DR-GCN), then propose a novel temporal message propagation network (TMP). Both the two proposed networks take the normalized graph G of a smart contract function as input, and output the label ˆ y ∈ {0, 1} indicating whether the function has a vulnerability of a certain type.

DR-GCN. [Kipf andWelling, 2017] proposes to apply convolutional neural networks to graph-structured data, which develops a layer-wise propagation network as:

where ˆ A = A+ I is the adjacency matrix (A) enhanced with self-loops (I), Xl is the feature matrix of layer l, and Wl is a trainable weight matrix. In the equation, the diagonal node degree matrix ˆD is used to normalize ˆ A. We first increase the connectivity between nodes in the normalized graph G by using the square of A. Then, we further take into account that the graph is already well normalized in our setting, and therefore remove matrix ˆD from the equation. Finally, we arrive at the solution: Xl+1 = σ ((A2 + I)XlWl ).

TMP. We also propose a TMP network, consisting of a message propagation phase and a readout phase (Fig. 2). In the message propagation phase, TMP passes information along the edges successively by following their temporal order. Then, TMP computes a label for the entire graph G by using a readout function, which aggregates the final states of all nodes in G. Formally, G = {V,E}, where V consists of all the major nodes and E contains all the edges. Denote E = {e1, e2, . . . , eN}, where ek represents the kth temporal edge.

Message propagation phase. Messages are passed along the edges, one edge per time step. At time step 0, the hidden state h0i for each node Vi is initialized with the feature of Vi. At time step k, message flows through the kth temporal edge ek and updates the hidden state of Vek, namely the end node of ek. Particularly, message mk is computed basing on hsk, the hidden state of the starting node of ek, and the edge type tk :

where ⊕ denotes concatenation operation, matrix Wk and bias vector b are network parameters. The original message xk contains information from the starting node of ek and edge ek itself, which are then transformed into a vector embedding using Wk and b. After receiving the message, the end node of ek updates its hidden state hek by aggregating information from the incoming message and its previous state. Formally, hek is updated according to:

where U, Z, R are matrices, while b1 and b2 are bias vectors. Readout phase. After successively traversing all the edges in G, TMP computes a label for G by reading out the final hidden states of all nodes. Let hTi be the final hidden state of the ith node, we may generate the prediction label ˆ y by

where f is a mapping function, e.g., a neural network, and |V | denotes the number of major nodes. However, we found that the differences between the final hidden state hTi and the original hidden state h0i are informative in the vulnerability detection task. Therefore, we instead consider to compute ˆ y as follows:

where ⊙ denotes element-wise product. Wj, b(1) j , and b(2) j , with subscript j ∈ {g, o} are model parameters to be learned.

Both the two networks DR-GCN and TMP are trained for contract vulnerability detection. During training, networks are fed with a large number of normalized graphs constructed from smart contract functions, together with their ground truth labels. Then, the trained models are employed to absorb a normalized graph and yield a vulnerability detection label. We would like to point out that we developed automation tools for converting source code to normalized graphs, therefore, the whole procedure is fully automated.

4 Experiments

4.1 Datasets and Experimental Settings

Datasets. Extensive experiments are conducted on all the smart contracts that have source code on the Ethereum and VNT Chain platforms. We denote the two real-world smart contract datasets as ESC (Ethereum Smart Contracts) and VSC (VNT chain Smart Contracts), respectively.

• ESC consists of 40,932 Ethereum smart contracts with roughly 307, 396 functions in total. Among the functions, around 5, 013 functions possess at least one invocation to call.value, making them potentially affected by the reentrancy vulnerability. Around 4, 833 functions contain the block.timestamp statement, making them susceptible to the timestamp dependence vulnerability.
• VSC consists of 4, 170 smart contracts collected from the VNT Chain ∗ , roughly containing 13, 761 functions. VNT Chain is an experimental public blockchain platform proposed by companies and universities from Singapore, China, and Australia.

Experimental settings. We compared our approaches (DR-GCN and TMP) with a total of twelve other methods, namely four existing smart-contract vulnerability detection methods (Oyente [Luu et al., 2016], Mythril [Mueller, 2017], Smartcheck [Tikhomirov et al., 2018], and Securify [Tsankov et al., 2018]), four neural network based methods (VanillaRNN, LSTM, GRU, and GCN), and four program loop detection methods (Jolt [Carbin et al., 2011], PDA [Ibing and Mai, 2015], SMT [Kling et al., 2012], and Looper [Burnim et al., 2009]). For each dataset, we randomly pick 20% contracts as the training set while the remainings are utilized for the testing set. In the comparison, metrics accuracy, recall, precision, and F1 score are all involved. In consideration of the distinct features of different platforms, experiments on reentrancy vulnerability and timestamp dependence vulnerability are conducted on the ESC dataset, while experiments on infinite loop vulnerability detection are conducted on the VSC dataset.

∗Vntchain website, 2018. https://github.com/vntchain/go-vnt.

4.2 Comparison with ExistingMethods

In this subsection, we first benchmark the proposed approaches (DR-GCN and TMP) against state-of-the-art methods on the reentrancy, timestamp dependence, and infinite loop vulnerabilities, respectively. Then, we compare our approaches with other neural network based methods.

Comparison on Reentrancy Vulnerability Detection

First, we compare our DR-GCN and TMP methods with state-of-the-art smart contract vulnerability detection methods, namely Oyente [Luu et al., 2016], Mythril [Mueller, 2017], Smartcheck [Tikhomirov et al., 2018], and Securify [Tsankov et al., 2018], on the reentrancy vulnerability detection task. The performance of different methods is presented in the left of Table 2, where metrics accuracy, recall, precision, and F1 score are engaged.

From the quantitative results of Table 2, we have the following observations. First, we find that existing tools have not yet achieved a satisfactory accuracy on reentrancy vulnerability detection, e.g., the state-of-the-art tool yields a 71.89% accuracy. Second, TMP outperforms state-of-the-art methods by a large margin. More specifically, TMP achieves an accuracy of 84.48%, gaining a 12.39% accuracy improvement over state-of-the-art tools. Besides, the F1 score of TMP is 24.54% higher than existing methods. Thirdly, DR-GCN also achieves better results than other existing methods in terms of all the four metrics. The strong empirical evidences reveal the great potential of applying graph neural networks to smart contract vulnerability detection.

Comparison on Timestamp Dependence Vulnerability Detection

We then compare the proposed methods with state-of-the-art smart contract vulnerability detection tools on the timestamp dependence vulnerability detection task. The comparison results are demonstrated in the middle of Table 2. The stateof-the-art method has obtained a 61.08% accuracy on timestamp dependence vulnerability detection, which is quite low. This may stem from the fact that most of existing methods detect timestamp dependence vulnerability by crudely checking whether there is block.timestamp statement in the function. Moreover, in consistent with the results on reentrancy vulnerability detection, TMP keeps delivering the best performance in terms of all the four metrics, while DR-GCN ranks the second. In particular, TMP gains a 22.37% accuracy improvement over state-of-the-art method.

We further look into the existing smart contract vulnerability detection tools to investigate the reasons behind the observations. Smartcheck fundamentally depends on a few rigid and simple logic rules to detect vulnerabilities, which leads to low accuracy and F1 score. Oyente employs data flow analysis to improve the accuracy, while its underlying patterns for detecting vulnerabilities are not so accurate. Regarding Mythril, it requires sophisticated techniques such as taint analysis or manual audit, which attains a medium accuracy. Unlike other methods, Securify classifies smart contract functions into violations, warnings, and compliances, where violation denotes the function is guaranteed to have the vulnerability (positive), and compliance denotes the function is safe (negative). We treat all warnings as negative since users are usually attracted by violations while ignoring a lot of warnings. Securify performs better than other existing methods, but has a high false negative rate.

Comparison on Infinite Loop Vulnerability Detection

For the infinite loop vulnerability detection, we compare our methods against available tools including Jolt [Carbin et al.,

2011], SMT [Kling et al., 2012], PDA [Ibing and Mai, 2015], and Looper [Burnim et al., 2009]. We empirically find that almost all existing methods fail to detect the infinite loop bug caused by the fallback mechanism of smart contracts. In contrast, our methods can successfully identify this vulnerability. This is because we explicitly model the fallback mechanism of smart contracts and consider data dependencies and control dependencies between program elements. Quantitative results are illustrated in the right of Table 2. From the table, we see that TMP consistently and significantly outperforms the other methods on the infinite loop vulnerability detection task. In particular, TMP and DR-GCN respectively achieve a 15.05% and 8.78% accuracy improvement over state-of-theart methods.

We further visualize the comparison results in Fig. 3(a), (c), and (e). Fig. 3(a) and Fig. 3(c) present comparison results of reentrancy vulnerability detection and timestamp dependence vulnerability detection, respectively. The 6 rows (in different colors) from front to back denote the Smartcheck, Oyente, Mythril, Securify, DR-GCN, and TMP methods, respectively. For each row in the figures, accuracy, recall, precision, and F1 score are respectively demonstrated from left to right. Fig. 3(e) shows comparison results of infinite loop vulnerability detection, where the the 6 rows from front to back denote the Jolt, PDA, SMT, Looper, DR-GCN, and TMP methods, respectively. We can clearly observe that DR-GCN and TMP outperform existing methods.

Comparison with Neural Network Based Methods

In order to seek out which neural network architectures could succeed in smart contract vulnerability detection, we also compare our methods with other neural network alternatives. Specifically, Vanilla-RNN, LSTM, GRU, and GCN are compared with our DR-GCN and TMP networks. For fair comparison, all the methods are presented with the vector representation of the normalized graph extracted from the source code and are required to detect the corresponding bugs. We illustrate the results of different models in terms of accuracy, recall, precision, and F1 score in Table 2. Fig. 3(b), (d), and (f) further visualize the results.

Interestingly, experimental results show that conventional recurrent neural networks Vanilla-RNN, LSTM, and GRU perform no better than existing vulnerability detection methods. In contrast, graph neural networks GCN, DR-GCN, and TMP, which are capable of handling graphs, achieve significantly better results than existing methods. This suggests that blindly treat the source code as a sequence is not suitable for the vulnerability detection task, while modeling the source code into graphs and adopting graph neural networks is promising. We conjecture that conventional recurrent models lose valuable information from smart contract code since they ignore the structural information of contract programs, such as the data-flow and invocation relationships.

We would like to highlight that the proposed TMP and DRGCN model consistently and significantly outperforms other neural network models in terms of all the 4 metrics. Besides TMP and DR-GCN, the GCN model performs the best. The accuracies of GCN and DR-GCN are lower than TMP. We attribute this to the fact that GCN fails to capture the temporal information induced by data flow and control flow, which is explicitly addressed in our TMP model using ordered edges.

4.3 Study on The Effect of Graph Normalization

By default TMP adopts the graph normalization module to highlight the major nodes in the graph, it is interesting to see the effect of removing this module. We removed the graph normalization phase from TMP and DR-GCN, and compared them with the default TMP and DR-GCN. The two variants are respectively denoted as TMP-WON and DR-GCN-WON, where WON is short for without normalization. Quantitative results are summarized in Table 3. We can see that with the proposed normalization module, the performance of both DR-GCN and TMP is better. For example, on the reentrancy vulnerability detection task, the DR-GCN model obtains a 4.39% and 4.04% improvement in terms of accuracy and F1 score, respectively, while TMP gains a 2.57% and 2.41% improvement in accuracy and F1 score.

Fig. 4 further plots the ROC curves of DR-GCN, TMP, and their variants. We adopt Receiver Operating Characteristic (ROC) analysis to show the impact of the graph normalization module. AUC (area under the curve) is used as the measure for performance, the higher AUC the better performance. Fig. 4(a) demonstrates that DR-GCN and TMP achieve better results on the reentrancy detection task with the normalization module, namely the AUC increases by 0.07 and 0.03, respectively. Regarding the timestamp dependence detection task, DR-GCN and TMP obtain a 0.05 and 0.03 improvement in AUC (shown in Fig. 4(b)). For the infinite loop detection task, DR-GCN and TMP gain a 0.08 and 0.05 improvement in AUC (shown in Fig. 4(c)). The experimental results suggest that program elements should contribute distinctly in vulnerability detection rather than having equal contributions.

5 RelatedWork

Smart contract vulnerability detection. Smart contract vulnerability detection is one of the fundamental problems in blockchain security. Current work mainly relies on symbolic execution methods, such as Oyente [Luu et al., 2016], Maian [Nikoli´c et al., 2018] and Securify [Tsankov et al., 2018], which suffer from high false negative rates due to the inability to explore all possible program paths. Recent work [Jiang et al., 2018] explores dynamic execution for vulnerability detection, but requires a hand-crafted agent contract for reentrancy detection, preventing it from fully automated application.

Graph neural networks (GNNs). With remarkable success of neural networks in various fields [Cheng et al., 2019] [Liu et al., 2018a], graph neural network has received increasing attention. Existing approaches roughly cast into two categories: (1) Spectral-based approaches generalize wellestablished neural models like CNNs for graph data. For instance, GCN [Kipf and Welling, 2017] implements a firstorder approximation of spectral graph convolutions [Defferrard et al., 2016], while [Li et al., 2018] proposes a graph CNN capable of processing input data of arbitrary graph structure. (2) Spatial-based methods inherit ideas from recurrent GNNs and adopt message passing for graph convolutions. [Micheli, 2009] directly sums up a node's neighborhood information for graph convolutions, while recent work [Veliˇckovi´c et al., 2017] and [Zhang et al., 2018] learn different weights of neighboring nodes using attention mechanisms.

6 Conclusion

In this paper, we have proposed a fully automated vulnerability analyzer for smart contracts. In contrast to existing methods, we explicitly model the fallback mechanism of smart contracts, consider rich dependencies between program elements, and explore the possibility of using novel graph neural networks for vulnerability detection. Extensive experiments show that our method significantly outperforms state-of-theart methods and other neural networks. We believe our work is an important step towards revealing the potential of deep learning methods on smart contract vulnerability detection tasks.

Acknowledgments

This paper is supported by the National Key R&D Program of China (2017YFB1401300, 2017YFB1401304), the National Natural Science Foundation of China (No. 61902348), the Natural Science Foundation of Zhejiang Province, China (Grant No. LQ19F020001).