Online Credit Payment Fraud Detection via Structure-Aware Hierarchical Recurrent Neural Network

1 Introduction

Recent year dramatically increased e-commerce payments have resulted in the booming occurrence of fraudulent transactions. It has been reported that $24.26 billion was lost due to digital payment fraud worldwide in 2018, which increased by 18.4% compared to 2017 and is still climbing 1 . Online credit payment fraud detection is therefore increasingly important to restrain the impact of fraud on the quality of services, costs and reputation of financial service institutions. Machine learning methods based on feature engineering have played an essential role in financial fraud detection [West and Bhattacharya, 2016; Abdallah et al., 2016], which depends on the effectiveness of the statistical features extracted from different aspects, such as user profiles and historical transactions [Bahnsen et al., 2016]. However, criminals may commit few transactions before making a fraudulent payment, or pretend to be a benign user to make multiple small normal payments and then make a large fraudulent payment. Under these scenarios, the statistical-based features may fail to effectively capture these users' fraudulent patterns, which results in a misjudgement [Zhong et al., 2020].

∗Corresponding author

1 https://shiftprocessing.com/credit-card-fraud-statistics/

Recently, RNN-based models show promising results by utilizing users' sequential behaviors that can timely reflect their dynamic and evolving intentions [Jurgovsky et al., 2018; Feng et al., 2019]. In these methods, actions of a user on a web page, e.g., click, search, view image, etc., are usually considered as his/her behavior sequence, and the intrinsic structure information of web pages are usually overlooked. A toy example of the intrinsic structure information of web page is demonstrated in Figure 1. In such a figure, a page consists of several blocks, and each block may contain multiple actions. The schema page→block→action is the intrinsic structure of web page used in this paper 2 . Multi-scale behaviors over different granularities can be derived given specific intrinsic web page structures, which might be beneficial to characterize the users' intentions. First, multi-scale behaviors may complement each other. For instance, Figure 2 demonstrates two multi-scale behavior sequences with the same action and block sequences. However, considering information in the page sequence, the potential frauder could be unearthed. Second, the natural tree-like subordinate relations of web page structure shed new light on aggregation multi-scale behaviors to capture hierarchical collaborative behavioral patterns. For example, action sequences in the same block may together be helpful to reflect user's intention in such a block.

2 Other schema is also alternative in our model.

To comprehensively exploit multi-scale user behavior sequences derived by web page structures, we propose a novel method coined Structure-Aware Hierarchical Recurrent Neural Networks (SAH-RNN for short) in this paper. First, we stack multiple RNN layers to consume multi-scale behavior sequences. Each layer models the corresponding behavior sequence on given granularity of web page structure. Then a subordinate relation guided update and interaction approach among RNN layers is devised. Basically, the upper layers which model compendious behaviors are updated less frequently and receive the summarized representations from lower layers. Furthermore, a dual attention mechanism is designed to perceive the impacts on both sequential information within the same sequence and structural information among different granularity of web pages.

The main contributions are summarized as follows:

• To the best of our knowledge, it is the first work to incorporate web page structures into behavior sequence modeling for online credit payment fraud detection task.
• We propose a novel SAH-RNN which employs stacked RNN layers equipped with the structural subordinate relation guided update strategy and dual attention mechanisms to perceive users' global and local intentions.
• Experimental results on a large-scale real-world dataset from Alibaba platform show that the proposed model significantly outperforms recent state-of-the-art approaches, and has a better generalization ability for various behavior sequence lengths.

2 RelatedWork

2.1 Credit Payment Fraud Detection

The credit payment fraud occurs when accounts or credit cards are stolen and paid. Conventional solutions usually extract statistical features from different aspects (e.g., user profiles, transaction summarizing and interaction relations) and detect fraudulent transactions with supervised learning algorithms, such as tree-based algorithms [Van Vlasselaer et al., 2015; Xuan et al., 2018] and neural networks [Fu et al., 2016; Liu et al., 2018; Zhang et al., 2018].

However, these methods aggregate historical transactions as a whole and ignore the drifting trend of users' interests. Recent works start to utilize the users' sequential behaviors to enrich the representation of users' dynamic and evolving interests. [Jurgovsky et al., 2018] employs the Long ShortTermMemory Networks (LSTM) to aggregate the users' historical purchase behaviors with the goal to improve fraud detection accuracy. [Babaev et al., 2019] presents EmbeddingTransactional Recurrent Neural Network (E.T.-RNN) to compute credit scores of the bank customers by examining their historical credit and debit card transactions. Apart from transaction sequence-based methods, the e-commerce platforms can capture more detailed behaviors in websites (e.g., click, search and edit operations), which have been proven effective for fraud detection in e-commerce scenario. For instance, [Wang et al., 2017] presents a fraud detection system in which the Stacked LSTM is used to capture the users' click and browsing behavior sequences. [Liu et al., 2020] proposes LIC Tree-LSTM that exploits local intention calibration for fraud transaction detection.

Comparatively, both the temporal and hierarchical representations of users' sequential behaviors are modeled in our work, which makes it distinct to the existing ones.

2.2 Hierarchical RNN

Hierarchical structures naturally exist in many sequential data. A typical approach to learn both hierarchical and temporal representation is to stack multiple recurrent layers vertically [El Hihi and Bengio, 1996; Chung et al., 2017], which called hierarchical RNN. Various hierarchical RNNs have been proposed to capture the latent hierarchical structure in sequences. For example, in the clockwork RNN (CWRNN) [Koutn´ık et al., 2014], the hidden layer is grouped into separate modules, each of which is interconnected and is explicitly assigned different timescales for update. Dilated RNN [Chang et al., 2017] is constructed by stacking multiple dilated recurrent layers with hierarchical dilations, which mitigates the vanishing gradient problem. Ordered Neurons LSTM (ON-LSTM) [Shen et al., 2019] introduces an elegant way of adding a hierarchical inductive bias to integrate the latent tree structures in sequences into recurrent models, achieving state-of-the-art performance on many sequential tasks.

Inspired by multi-scale behavior sequences that are derived according to the natural tree-like structure of web pages, we propose to incorporate the structural information into hierarchical RNN architecture, which has not been explored before.

3 Methodology

3.1 Problem Statement

In this paper, the online credit payment fraud detection task is phrased as a binary sequence classification problem. That is, given the user's multi-scale behavior sequence X = [X1;X2; ...;XT ] before initiating a payment (also known as a transaction), where T is the length of behavior sequence, Xt ∈ RL∗d denotes the multi-scale behaviors at the t-th time step and L is the number of structure hierarchies, our purpose is to predict whether it is a fraudulent transaction. We assign a label y ∈ {0, 1} on each transaction to indicate whether it is fraud or not. Hence, given the training set D = {(X, y)}, our goal is to predict the fraud probability of each transaction in the testing set.

3.2 Model Overview

In the following, we present an SAH-RNN variant based on LSTM, called Structure-Aware Hierarchical LSTM (SAHLSTM). As shown in Figure 3, the overall architecture has three main components, namely Structure-Aware Factor Sequences Extraction, Hierarchical RNN Layers and Dual Attention Mechanism. The structure-aware factor sequences are extracted from the multi-scale behavior sequences to determine the timing of updating and interacting for the hierarchical RNN layers. The hierarchical RNN consists of multiple stacked RNN layers. Each layer models the corresponding behavior sequence at the given granularity of web page structure and interacts with other layers when the update operation is executed. The dual attention mechanism including sequence attention and structure attention layers are designed to capture the users' global and local intentions simultaneously.

Structure-Aware Factor Sequences Extraction

To automatically capture the structure information of web page, the structure-aware factor sequences S ∈ RL∗T are extracted from the multi-scale behavior sequences X, slt stands for the structure-aware factor in the l-th layer at the t-th time step, which is calculated as follows.

The binary value of slt indicates whether the behavior in the next time step varies from the current time step. As the upper layers have more repeated behaviors, the binary sequences S are getting sparse from lower layers to upper layers. Moreover, the structure-aware factor sequences determine whether to update the hierarchical RNN layers and are used as the masking vector of the sequence attention layer.

Hierarchical RNN Layers

Taking the multi-scale behavior sequences and structureaware factor sequences as input, the depth of the stacked LSTM layers is the same as the number of structure hierarchies, and each layer models a behavior sequence at the corresponding granularity from fine to coarse. At each time step, each layer executes either update operation or copy operation, which is determined by the structure-aware factor slt . The update function for the hidden states h and cell states c at each layer l is defined as follows.

where f, i, o, g are forget gate, input gate, output gate and candidate vector, respectively, and ⊙ is the element-wise product. Note that unlike the standard stacked LSTM which are updated at every time step, the upper layers are updated less frequently than the lower layers. It is because the upper layer models the behavior sequence with a coarser granularity. The update operation is executed only when the behavior in the next time step varies from the current, and f , i, o, g are obtained by the following equations.

where σ is the sigmoid function, W and b denote the weight matrix and bias vector, respectively. ˜xt is the input to this step, which is defined as follows.

At the first layer, the input is the embedding of the finestgrained behavior sequence x1t . Otherwise, we concatenate xlt with the hidden state of the lower layer hl-1 t as the input to the upper layer l. This operation allows the upper layer to absorb the summarized representation of behaviors at the lower layer. Thus, it is reasonable to expect that the upper layer obtains a more comprehensive global view, benefiting from the memorization of long-term dependencies based on short-term dependencies learned from the lower layer.

Sequence Attention Layer

Considering that behaviors at the same LSTM layer are strongly related to each other, and there are many consecutive repeated behaviors exist in the upper layer which are not updated at every step. We employ the multi-head selfattention mechanism [Vaswani et al., 2017] to capture the inner relationship between behaviors at the same layer and decrease the effect of repeated behaviors at the upper layers. Specially, the self-attention operations are applied to the hidden states of each LSTM layer separately. Mathematically, let Hl = [hl1 ; hl2 ; ...; hlT ] ∈ RT∗d be the input of the l-th selfattention layer, and the i-th head is calculated as follows.

where the projection matrices W Ql ,W Kl ,W Vl ∈ Rd∗d . As for the input of the softmax function, we use the structureaware factor sequence at the corresponding layer of S as the masking vector to mitigate the impact of consecutive repeated behaviors on the upper layer. Then the concatenated vector of different heads is fed into the feed-forward network:

where FFN(·) is the feed-forward network, W Ol ∈ Rhd∗d is the projection matrix, and h is the number of heads. Residual connection and layer normalization are successively conducted around both the multi-head attention and feed-forward network. Then the distilled representation of F l is calculated as follows.

where Avg(·) is the average pooling.

Structure Attention Layer

We assume that the infrequently updated upper layers have a more global view, while the frequently updated lower layers can better capture short-term intentions. Intuitively, different layers are likely to have different importance for the prediction task. We thus devise a structure attention mechanism to merge all representations of behavior sequences F ′1 , F ′2 , ..., F ′L into a distilled representation. The definitions of a structure attention mechanism are as follows.

where Wa and ba are the trainable weight matrix and bias, respectively. al is the attention weight for the corresponding behavior sequence, and z denotes the final representation of multi-scale behavior sequences generated by SAH-LSTM.

Model Training

The obtained final representation z is then fed into a fully connection layer with a sigmoid unit, and the predicted fraud probability is calculated as follows.

where wp and bp are the weight vector and bias, respectively, and p is the predicted fraud probability. Finally, our model is trained with the negative log-likelihood function, which is defined as follows.

where D is the training dataset, the input X is the users' multi-scale behavior sequences, and y is the ground truth.

4 Experiments

4.1 Dataset

To verify the effectiveness of SAH-RNN in the real-world industrial applications, we conduct extensive experiments on a large-scale dataset 3 from Alibaba platform (www.alibaba. com), one of the most popular e-commerce platforms in China. According to the general definition in financial scenarios, we define the positive samples as the fraudulent transactions and the negative samples as genuine transactions. The details of the dataset is summarized in Table 1. For the behaviors over different structure granularities, we did some necessary preprocessing like clustering, and selected top frequent behavior categories as our vocabulary. The detailed statistical information are shown in Table 2.

3 only includes website browsing behaviors permitted by the user.

4.2 ComparedMethods

(1) Sequence-based Methods

• LSTM is a widely used RNN variant [Hochreiter and Schmidhuber, 1997]. Here we consider both single-layer LSTM and 3-layer Stacked LSTM.
• GRU simplifies the gates in LSTM [Chung et al., 2014]. Both single-layer and 3-layer Stacked GRU are compared.
• TextCNN uses multiple one-dimensional convolutional layers with different kernel sizes to capture the dependencies among behaviors [Kim, 2014].
• Transformer is an essentially attention based model [Vaswani et al., 2017].

(2) Hierarchical RNN-based Methods

• CW-LSTM separates the hidden layer into several modules for processing inputs at different temporal granularity to better capture long-term dependencies [Koutn´ık et al., 2014].
• Dilated LSTM is constructed by stacking multiple recurrent layers with multi-resolution dilated recurrent skip connections [Chang et al., 2017]. We consider 3layer Dilated LSTM for comparison.
• ON-LSTM integrates tree structures into recurrent neural networks by ordered neurons to obtain hierarchical representations [Shen et al., 2019]. We consider 3-layer ON-LSTM for comparison.

4.3 Implementation Details

All the models are implemented with Tensorflow [Abadi et al., 2016]. For fair comparisons, we conduct experiments with different configurations on the number of layers and hidden units in various compared models. For each transaction, we backtrack the user's behaviors in the last 7 days before making payment. We limit the maximal length of the behavior sequence to 500 and paddings are performed for too short behaviors. For all the experiments, we under-sampled the negative examples to lift the ratio of positive samples (fraud transactions) at 10% in the training dataset. We randomly extract 10% samples from the original training set for validation, and perform early stopping if the validation performance is not improved for 10 epochs. Moreover, we choose Adam [Kingma and Ba, 2015] as optimizer and decide the initial learning rate from {0.01, 0.001, 0.0001} via validation. We set the batch size to 512 and deliberately optimize the parameters in other compared methods according to their literatures.

4.4 Metrics

Since the positive rate in the online payment scenario is really low in general, we evaluate the performance of the approaches with AUC and R@P0.1. The AUC is the area under the ROC curve, which reflects the ranking ability. The R@P0.1 is the recall when the precision equals to 10%, which indicates the ability to detect top-ranking positive samples. In summary, the higher both two evaluate metrics are, the better performance the results have.

4.5 Main Results

Table 3 demonstrates the main results. The major findings from the results can be summarized as follows:

1. Compared with the single-layer RNN variants, our proposed model facilitates the hierarchical RNN architecture to fully exploit multi-scale behavior sequences, and we can clearly observe that our model significantly outperforms single-layer LSTM or GRU by a large margin, e.g., SAHLSTM outperforms GRU with about 4.00% increased AUC and 8.42% increased R@P0.1. Meanwhile, the usage of different update frequencies at different layers makes our models more advantageous than stacked RNN variants. For instance, SAH-LSTM gets 3.24% higher AUC and 6.04% higher R@P0.1 than 3-layer stacked LSTM. Furthermore, SAH-LSTM is also more advanced than the CNN-based and attention-based methods, i.e., TextCNN and Transformer, with 6.41% and 3.44% increased R@P0.1, respectively.

(2) The explicit boundary information extracted from the structure of web page enables SAH-LSTM to achieve a better performance than other hierarchical RNN models. CWLSTM and Dilated LSTM update each layer with a fixed but different rate without considering that non-stationarity is prevalent in behavior sequences. ON-LSTM dynamically adapts the update frequencies for different neurons, and its performance is better than CW-LSTM and Dilated LSTM, but still worse than SAH-LSTM. These observations demonstrate that our proposed models are more effective than other hierarchical RNN models due to taking advantage of structural information and the dual attention mechanism.

(3) Among these baseline methods, we can find that ONLSTM performs the best on AUC metric while Transformer is the best on R@P0.1 metric. We conjecture their improvements mainly come from the consideration of the global intention. In more detail, Transformer treats the entire sequence as a whole and utilizes the self-attention mechanism to model the interactions within the sequence. ON-LSTM, on the other hand, adopts the high-ranking ordered neurons to encode long-term information. However, without the consideration of structural information among multi-scale behavior sequences, they can hardly maximize the performance when the sequence is short, and it is the key factor that SAH-LSTM can outperform these two methods.

4.6 Ablation Test

We perform the ablation study and the results are shown in Table 4. Specifically, we compare SAH-LSTMwith its three variants, namely SAH-LSTM\SeqAtt (without sequence attention), SAH-LSTM\StructAtt (without structure attention) and SAH-LSTM\LayerInter (without layer interaction, i.e., the upper layers will not receive the summarized representation from the lower layers). The results show that both AUC and R@P0.1 get worse by removing any of these three designs, which reflects that all three designs are important. The sharply decreased R@P0.1 in SAH-LSTM\SeqAtt and SAHLSTM\StructAtt indicates the effectiveness of the dual attention layers. The SAH-LSTM\LayerInter performs worse than other variants, which illustrates that the information transmission between RNN layers has a more significant impact on detecting fraudulent transactions in our dataset.

4.7 Further Discussion

Impact of Behavior Sequence Length

We divide the testing set into 5 groups by the interval of 100 to analyze the impact of different behavior sequence lengths, as shown in Table 3. We further put the sequences whose length more than 500 as a group separately, and the AUC and R@P0.1 curves are shown in Figure 4. The results indicate that all models' performance improve obviously with the increase of behavior sequence length at the beginning. However, the performance of the other compared baseline methods begins to deteriorate as the sequence length further increases, while the curves trend of SAH-LSTM are flatter, and achieve better results on both shorter and longer sequences than other compared models. It gives us an encouraging result that our proposed models can better distinguish the fraud patterns for both new customers with few behaviors and mature customers with more complex behaviors.

Effect of Structure Attention Layer

We next visualize the distribution of structure attention weights for the transactions in testing set, as shown in Figure 5. It can be observed that the fraudulent transactions have higher attention weights on action-level behaviors, which are modeled by the frequently updated lower layer and can better capture the local mutational behavior patterns. While the benign transactions pay more attention to page-level behaviors which have a more comprehensive view. We conjecture the possible reason for these phenomena is the structure attention layer can perceive users' global and local intentions.

5 Conclusion

In this paper, we proposed an SAH-RNN to model multi-scale behavior sequences derived by web page structures. The subordinate relations of web page structures guided the update strategy of the network. A dual attention mechanism was devised to learn interactions among the multi-scale behaviors and simultaneously capture users' global and local intentions. Experimental results on the large-scale real-world dataset demonstrate the effectiveness of the proposed model.

Acknowledgements

This work is supported by Alibaba Innovative Research Program. The work is also supported by the National Natural Science Foundation of China under Grant No.92046003, 61976204, U1836111, the Project of Youth Innovation Promotion Association CAS and Beijing Nova Program Z201100006820062.