Adapting Meta Knowledge with Heterogeneous Information Network for COVID-19 Themed Malicious Repository Detection

1 Introduction

Along with the global health crisis, the cyberattacks have increased by more than 250% and caused over $300 million loss in 2020 [Waggoner and Markowitz, 2021]. During the pandemic, using malware as a major weapon, cybercriminals utilized coronavirus disease (COVID-19) related information as a lure to infiltrate victims' systems to perform attacks such as stealing users' virtual assets and compromising network, which have caused significant loss of individuals, corporations, and governments. To perform the attacks, cybercriminals may post malicious code repositories with a COVID-19 theme in social coding platforms to disseminate malware. As GitHub has become the largest source code repository platform with 40 million users and over 280 million public repositories, it has been utilized by cybercriminals to disseminate malware to compromise security of the cyberspace [Fan et al., 2019]. For example, a COVID-19 themed repository with embedded malicious code hosted in GitHub could be directly forked by other developers and easily disseminated through other social platforms like Reddit. Therefore, preventing the propagation of COVID-19 themed malware is in urgent need. In this paper, we solve this problem by developing an intelligent framework to detect COVID-19 themed malicious repositories in GitHub.

∗Corresponding authors.

To detect malicious code repositories, existing methods [Semmle, 2019a; Semmle, 2019b; Rokon et al., 2020] mainly rely on code content while ignoring the rich structured relations among entities in social coding platforms, impairing the effectiveness for judging their legitimacy. More specifically, GitHub brought in two vulnerability detection products CodeQL [Semmle, 2019a] and LGTM [Semmle, 2019b] in 2019, which can automatically analyze the variants of critical vulnerabilities based on code content of the repository. Additionally, although some effective models [Zhang et al., 2020a; Fan et al., 2019] for malicious code repository detection have been proposed, they require sufficient samples for model training. For example, Rokon et al. [Rokon et al., 2020] utilized 97K labeled repositories in GitHub to train a supervisedlearning model for malicious repository detection. Thus, existing models are not suitable for detecting novel types of malicious repositories with few data, such as malware with a COVID-19 theme and their variants.

In this paper, we aim to address the above challenges by developing a novel model called Meta-AHIN (Figure 1(a)), which integrates graph convolutional neural network and meta-learning for COVID-19 themed malicious repository detection with few data constraint in GitHub. First, we construct an Attribute Heterogeneous Information Network (AHIN) to model the relationships among four types of entities (i.e., repository, keyword, file, and user) in GitHub. Next, we build an attention-based graph convolution network (AGCN) to fuse both structural relations and semantic content information for learning node representations in AHIN. To exploit and capture unlabeled information in AHIN, we further design a self-supervised module and refine the node embeddings which are fed to downstream classifier for malicious repository detection. Since novel types of malicious repositories with a COVID-19 theme may only have few data samples, we build a meta-learning framework to transfer knowledge from detection tasks of regular repositories and adapt them quickly for new detection tasks (e.g., COVID-19 themed malicious repository detection). In addition, considering different malicious repository detection tasks contribute differently to the meta-learner, we introduce task-aware attention weights to measure their importances. Self-supervised module and task weights are incorporated into AGCN and meta-learning respectively for better detection performance. To summarize, the major contributions of this work include:

• We solve the problem of COVID-19 themed malicious repository detection with few data constraint in GitHub, which is novel and important.
• To handle structural relations and unstructured content information in GitHub, and to account for the constraint of few labeled samples of COVID-19 themed malicious repository, we develop a novel model called Meta-AHIN by integrating graph convolutional network and meta-learning.
• We collect COVID-19 themed malicious repository data from GitHub and conduct extensive experiments using this dataset. Promising results demonstrate the effectiveness of our model by comparison with state-of-the-art methods.

2 RelatedWork

Malware Repository Detection. The recent studies for malicious repository detection usually focus on the analysis of code (or app) content [Ye et al., 2017; Ragkhitwetsagul et al., 2019; Meli et al., 2019; Rokon et al., 2020]. For example, Rokon et al. proposed a content-based supervised learning method to identify malicious repositories in GitHub [Rokon et al., 2020]. These methods, however, ignore the rich structural information among different types of entities (e.g., repository, user) in social coding platforms. Different from existing works, we study the characteristics of code repository by incorporating both structural relation and unstructured content information in GitHub.

Graph Convolutional Network (GCN). Our graph representation learning framework is conceptually inspired by GCN [Wu et al., 2020]. There are two main types of GCN: spectral methods and spatial methods. For spectral methods, they aim to represent nodes in graph and perform convolution in the spectral space [Bruna et al., 2013; Henaff et al., 2015; Defferrard et al., 2016]. For spatial methods, they operate the original graph directly and define convolution layers over nodes to aggregate information of local neighbors [Hamilton et al., 2017; Veliˇckovi´c et al., 2018]. Besides homogeneous GCN, some models [Zhang et al., 2019; Zhang et al., 2020b; Ye et al., 2020b; Ye et al., 2020a] handle heterogeneous graphs. Inspired by these studies, we build an attention-based GCN to learn node embeddings in AHIN.

Meta-Learning. Recent meta-learning models generally fall into two groups: (1) metric-based meta-learning [Vinyals et al., 2016; Snell et al., 2017], and (2) gradient-based metalearning [Finn et al., 2017; Lee and Choi, 2018; Finn et al., 2018]. For the former ones, they implement a generalized metric and matching functions from training tasks to train the model; for the latter ones, they propose to employ data of existing tasks to learn well initialized model parameters that can be updated to new tasks in a fast manner with few data. In this paper, we are motivated to use gradient based meta-learning to address the challenge of few labeled data for COVID-19 themed malicious repository detection.

3 Preliminaries

3.1 ProblemDefinition

Let G = (V, E, X) denote an attributed heterogeneous information network (AHIN), where V is the set of different types of nodes, E ⊆ V×V is the set of different edges, and X is the attributed feature set. A node can be regarded as any type of entities and an edge can be regarded as any type of relations between two nodes. There are four types of nodes (repository, keyword, file, and user) and seven types of relations (e.g., file-contain-keyword) in our built AHIN (see Figure 1(b)). The goal is to learn repository embeddings f which can be fed to a classifier for malicious repository detection. Given the features of all repository nodes Xr = (x1, . . . , xN) (N: number of repositories) and their labels Y = (y1, . . . , yN) (yi = 1 denotes malicious and yi = 0 represents benign), we aim to learn a mapping function Uφ : Xr → Y (with parameter φ). Unlike existing works that use sufficient samples for model training, we consider a more practical scenario that only few labeled data are available since novel COVID19 themed repositories only have limited malicious samples. Specifically, given different types of regular repositories and their labels, we aim to build a classifier which can be quickly adapted to predict labels of different types of COVID-19 themed repositories that are unseen during the model training, given few malicious samples of these repositories. Formally, the problem is defined as follows.

Problem 1. COVID-19 themed malicious repository detection. Given a set of different types of code repositories (w/o a COVID-19 theme) along with their attributed features Xr = (x1, . . . , xN) and corresponding labels Y = (y1, . . . , yN) as well as the built AHIN G (training data), the problem is to build a machine learning model to detect malicious code repositories with a COVID-19 theme that have few malicious samples (testing data).

3.2 Graph Convolutional Network

Since GCN has been proved to be powerful for learning the node representations [Kipf and Welling, 2017] by exploring both structure of a graph G and node features X, we employ GCN as the base model to learn the repository representations in AHIN. Our goal is to learn the embedding fi ∈ Rd (d: embedding dimension), which corresponds to the representation of repository node i. The layer-wise propagation rule of GCN is defined as follows:Hl+1 = σ( ˜ A Hl W l ), (1)

where Hl+1 denotes the node representations at l +1 layer andH0 =X, ˜ A is a symmetric normalization of A with selfloop, i.e., ˜ A = ˆD- 1 2 ˆ A ˆD- 12 with ˆ A = A+ IN. A, IN, ˆD are the adjacency matrix, the identity matrix, and the diagonal node degree matrix of ˆ A respectively. W l denotes the weight matrix at l-th layer, and σ is the activation function. For simplicity, we use f = GCN(X, A) to denote a GCN model.

4 ProposedModel

In this section, we present the details of Meta-AHIN. At first, we construct an AHIN to depict both structural relations and unstructured content information in GitHub. Then we propose an attention-based GCN which is further augmented by a self-supervised module, to learn the node embeddings for malicious repository detection. Lastly, we introduce a task attentive meta-learning procedure for model optimization.

4.1 AHINConstruction

To comprehensively describe repositories in GitHub, besides content-based features, we also consider semantic context and structural relation information in GitHub. As shown in Figure 1(b), we build an AHIN with four types of entities attached with content features and seven types of relations, such that all these information can be exploited simultaneously.

Content Feature. For each type of node in AHIN, we extract different content features (as listed in Table 1) and concatenate them to denote feature vector of the node. Note that for text content (e.g., readme), we merge all text information for each entity and feed them into the pre-trained BERT language model [Devlin et al., 2018] to acquire the corresponding feature vector. For instance, for keyword feature, we extract all of the keywords from every source code file to represent the repository. Then, we select a set of maliciousoriented keywords based on word frequency, and feed the keyword set of each repository to BERT to obtain a keywordbased feature embedding. For enumerated content (e.g., file type such as executable file), we apply one-hot encoding to convert it to a binary feature vector.

Relation. Besides content features, relations among different entities are also indispensable to judge the legitimacy of repository. For instance, a repository is more likely to be judged as malicious if it is owned by a cyber attacker (e.g., R3: repository-owned-by-user). Hence, we leverage seven types of edges (R1-R7 in Table 1) to describe the informative and complex relationships among four types of entities.

4.2 AHINRepresentation Learning

Attention-based GCN. After constructing the AHIN, we design an attention-based GCN to learn the repository embedding. Specifically, we first transform the input attribute features of all entities to a common space. For each node vi ∈ V with type Ti and its attributes Xi, the transform function g(X) is defined as follows:

where WTi is the transform weight matrix for node type Ti (|T| = 4). Then, considering the converge of node features with Laplacian smoothing [Kipf and Welling, 2017], we use a two-layer GCN, where the node embedding is formulated:

To handle the heterogeneous graph with different types of nodes, we then leverage attention mechanism to capture different influence of neighbors for generating node embeddings. Given two neighboring nodes vi, vj ∈ V , we define the effect coefficient between them via the Softmax function:

where Nvi denotes the neighbor set of vi. Then, the attentionbased embedding for vi is given as follows:

Further, for each repository node vr ∈ V, we feed its embedding f att r into a fully-connected layer to obtain its malicious score, i.e., ˆ Y att r = f att r W att . The objective is to minimize the cross-entropy loss between the given labels Y and the prediction score ˆ Y att :

where Vr is the set of labeled repository nodes in AHIN and Y is the label of repository (malicious or benign).

Self-Supervised Augmentation. In order to exploit the unlabeled information in AHIN, we further introduce a selfsupervised module to enhance the representation learning ability (see Figure 1(c)). Specifically, we randomly mask type attributes of some nodes in AHIN by using special masked indicators. Then, we employ a mean pooling layer AGGMEAN to aggregate neighboring nodes information of the masked nodes and obtain their embeddings:

where Nvk is the neighbor set of vk. Moreover, for each masked node vk, we feed f m k into a fully-connected layer, i.e., ˆ Z m k = f m k W m, to obtain the predicted value of node type. The objective of this module is to minimize the crossentropy between the given node types Z and the predictions:

where Vm is the set of masked nodes. Finally, the joint objective is defined as the combination of repository label classification and self-supervised masked node type prediction:

4.3 Task-aware Meta-Learning

MAML. We design the optimization strategy based on Model Agnostic Meta-Learning (MAML) [Finn et al., 2017]. Specifically, it applies gradient-based algorithm that learns well initialized model parameters which can be quickly adapted to unknown new tasks. In this paper, we are given a set of tasks T defined as classification tasks on different types of repositories. We use the classification tasks of code repositories (w/o a COVID-19 theme) as training tasks while those of COVID-19 themed repositories as testing tasks. Data sampled from each task τ ∈ T is divided into support set Sτ and query set Qτ . Then, the classifier is first updated to taskspecific model (w/o a COVID-19 theme) using support set Sτ , and further optimized to task-agnostic model using Qτ of all tasks in training data. After sufficient training, the learnt model further adapt to new tasks with few data samples in support set, which is called meta-testing. Let φ be the set of model parameters. For a certai task τ, α is the step size, we firstly feed Sτ to the model and calculate the loss Lτ on Qτ to update parameters φ to φ′τ through gradient descent:

Task-aware Attention. Unlike traditional meta-learning methods [Finn et al., 2017] treating each training task with equal weight for optimizing the meta-learner, we introduce the self-attention [Lee et al., 2019] to reflect the importance of different tasks (see Figure 1(d)). Particularly, the task attention weight is computed as follows:

where Fτ = MEAN (fτ ) represents the task embedding which is averaged by all of repository embeddings of τ . Then, with the task-specific model parameter φ′τ obtained from Eq. 10, the model parameters are updated as follows:

where β is the learning rate, and Lτ is the loss on Qτ . The Meta-AHIN procedure is illustrated in Algorithm 1.

Algorithm 1 Training Procedure of Meta-AHIN

Require X, A: nodes features and adjacent matrix of AHIN

• 1: Obtain repository embeddings f att and masked node embeddings f m via Eq. 3 and Eq. 7 respectively
• 2: Compute the joint objective L via Eq. 9
• 3: while not convergae do
• 4: Sample a batch of training tasks τ from T
• 5: for each τ do
• 6: Sample a support set Sτ and a query set Qτ
• 7: Update parameters via Eq. 10
• 8: end for
• 9: Calculate the importance of each task via Eq. 11
• 10: Update model parameters via Eq. 12
• 11: end while

5 Experiments and Analysis

In this section, we first introduce our collected data. Then we conduct extensive experiments to evaluate the performance of our proposed model and show related analysis.

5.1 Data Collection

We evaluate our model with the dataset crawled from GitHub. The dataset is divided into two groups: code repository with and w/o a COVID-19 theme. we crawl the open source repositories in GitHub from Feb 2020 to Dec 2020. Particularly, for COVID-19 themed repository, we crawl them based on a set of COVID-19 related keywords (e.g., coronavirus) and the corresponding user profile. There are seven types of repositories based on the application scenarios (e.g., tracking application). Four of them are regular repositories viewed as training tasks and the other three are COVID-19 themed repositories viewed as testing tasks. To obtain the ground truth, we apply a two-step mechanism: (i) we implement VirusTotal [Virustotal, 2017] having over 70 anti-malware scanning tools to validate the legitimacy of repository; and then (ii) we ask anti-malware experts to confirm the legitimacy of repositories. Hence, we obtain 20,895 repositories including 6,965 malicious (835 with a COVID-19 theme), 13,930 benign repositories (1,670 with a COVID-19 theme). Specifically, we construct an AHIN with 174,661 nodes (i.e., 20,895 repository nodes, 52,530 keyword nodes, 82,451 file nodes, and 18,785 user nodes) and 696,464 edges of R1-R7.

5.2 Baseline Methods

We compare our model with following baseline methods:

• Traditional Classification. We take the attributes of each repository as feature vector and feed it to logistic regression (LR) or 5-layer deep neural network (DNN) (B1).
• Meta-Learning. For meta-learning baseline methods, besides MAML [Finn et al., 2017], we also employ
• two popular few-shot learning models, Matching Network (Matching) [Vinyals et al., 2016] and Prototypical Network (Protonet) [Snell et al., 2017] to train a meta-learner to classify COVID-19 themed repository (B2).
• Graph Embedding. We also utilize four popular graph representation learning models to learn repository embeddings in AHIN: Deepwalk [Perozzi et al., 2014], metapath2vec [Dong et al., 2017], GCN [Kipf and Welling, 2017], and GAT [Veliˇckovi´c et al., 2018]. Then the learned repository embeddings are fed to a generic 5-layer DNN (B3). In addition, we also implement MAML (B4) framework to train these four models.

5.3 EvaluationMetrics and Parameter Settings

To evaluate the performances of our model and baseline methods, we adopt two widely-used metrics: F1 score (F1) and accuracy (ACC). The experiments are conducted under the environment of the Ubuntu 16.04 OS, plus Intel i9-9900k CPU, GeForce GTX 2080 Ti Graphics Cards, and 64 GB of RAM. For meta-learning models, inner-level and outer-level learning rate are set as 0.003 and 0.001 respectively. The update steps for the meta-testing task is 2. For graph representation learning models: the dimension of node embedding is 256. We apply Pytorch to implement all methods and use 10-fold cross-validations.

5.4 Comparison with Baseline Models

The performances of all models are reported in Table 2, where the best results are highlighted in bold and the best baseline results are indicated by underline. The shot number denotes the support data size. According to this table, we can find that (i) The performance of meta-learning models (with repository feature (B2) or node embedding (B4)) are better than traditional feature based classification models (B1). Particularly, MAML outperforms Protonet and Matching. (ii) For graph representation learning models (B3 and B4), GCN has the best performance when the support data size is small, while GAT has the best performance when the support data size is larger. (iii) In all cases, Meta-AHIN significantly outperforms all baseline methods for COVID-19 themed malicious repository detection, demonstrating the effectiveness of our model design. (iv) To further show performance for each type of COVID-19 themed malicious repository, we report results of our model and the best baseline methods in Table 3. It is easy to find that Meta-AHIN is robust for different tasks and significantly outperforms the baseline models.

5.5 Ablation Studies

Since Meta-AHIN integrates three essential components (i.e., attention-based GCN (AGCN), self-supervised module (Ssup), and task-attention MAML (taMAML)), we conduct extensive ablation studies to analyze the contributions of different components by removing each of them independently (see Table 4). Specifically, we remove MAML (A1) and task-attention (A2) from our model respectively. We can see that both (especially A1) results drop significantly, showing that both MAML and task-attention have large contribution to Meta-AHIN. In addition, we remove self-supervised module (A3) and neighbor attention of GCN (A4) form our model. We found that Meta-AHIN is better than A3 and A4, demonstrating that both self-supervised learning and neighbor attention are effective in enhancing the model.

5.6 Comparison with Industrial Methods

We further validate the performance of Meta-AHIN by comparison with other industrial security products including LGTM (provided by GitHub) and other popular anti-malware products integrated in VirusTotal [Virustotal, 2017] (i.e., DrWeb, McAfee, and Avast). From Table 5, we observe that Meta-AHIN achieves around 50% and 20% accuracy improvement in comparison with LGTM and Avast respectively. It shows that our model can significantly improve the performance of COVID-19 themed malicious repository detection.

5.7 Embedding Visualization

To better show the effectiveness of our model, we visualize embeddings of one type of repositories generated by MetaAHIN and GAT with MAML in Figure 2. The green and orange points represent the embeddings for benign and malicious repositories respectively. In Figure 2, we observe that our model splits benign and malicious repositories better than GAT+MAML, which shows the superiority of our model.

6 Conclusion

In this paper, we develop a novel model (i.e., Meta-AHIN) to solve the problem of COVID-19 themed malicious repository detection in social coding platform (i.e., GitHub) accounting the constraint of few labeled samples. In particular, our proposed Meta-AHIN employs an attention-based GCN to learn repository embeddings in AHIN constructed based on GitHub data. To exploit unlabeled information in AHIN, a self-supervised module is proposed and incorporated into the model. Moreover, a task attention-based meta-learning framework is developed to address the challenge of few labeled samples and to optimize model parameters. The experimental results based on the real-world dataset from GitHub demonstrate the effectiveness of our proposed model.

Acknowledgments

Y. Qian, Y. Zhang and Y. Ye's work is partially supported by the NSF IIS-2027127, IIS-2040144, IIS-1951504, CNS2034470, CNS-1940859, CNS-1814825, OAC-1940855 and ECCS-2026612, the NIJ 2018-75-CX-0032.