Gated POS-Level Language Model for Authorship Verification

1 Introduction

With a set of documents written by a known author, how can we determine whether another document is written by this author? This is the authorship verification problem [Koppel and Schler, 2004], which has important applications in broad domains, e.g., to detect fraud or phish in cybersecurity, to detect plagiarism in research, and to assist forensic investigation for the judiciary.

Numerous authorship verification methods have been introduced to tackle this problem [Stamatatos, 2009]. Most of these approaches focus on finding features that can effectively characterize writing styles [Abbasi and Chen, 2008]. Character level tri-grams is shown to be effective for authorship analysis [Bevendorff et al., 2019b]. There are also works trying to address this problem from the perspective of compression [Khmelev and Teahan, 2003; Halvani et al., 2017] and the distribution of function words [Koppel and Schler, 2004]. [Zhao et al., 2006] demonstrates that the POS tag is effective to characterize the writing styles from the perspective of syntax. However, they only use simple distributions to represent the pattern of each author, which is insufficient to capture the local correlations in the sequence.

∗Contact Author

Recently, deep learning has achieved exceptional successes in many natural language processing problems. However, applying deep learning on the authorship verification problem faces a major challenge: the amount of training texts for each author is extremely limited, which poses significant challenges to the training of neural networks that have a large number of parameters.

[Bagnall, 2015] is one of the few in applying deep learning to the authorship verification problem. They attempt to address the above challenge by building a character-level language model with a shared RNN layer and a separate output layer for each author to encode the author-specific writing styles. Their method sits at the first place in the PAN-15 authorship verification competition and demonstrates the great potential of deep learning for authorship verification. However, we find experimentally that character-level RNN has difficulties in learning syntactic information with a limited number of samples. In addition, although their model uses a shared RNN layer, the separate fully-connected output layers for each author have to learn from scratch independently, which may lead to sub-optimal language models. This also makes it difficult for their model to distinguish author-specific writing styles from common writing patterns shared by everyone.

In this paper, we introduce a novel POS-level gated language model to capture the author-specific syntactic writing styles in a data-efficient way. As shown in Figure 2, we utilize a POS tagger pre-trained on a large external data set to convert the raw inputs into sequences of POS tokens. The external author-agnostic syntactic information comes with the POS tagger limits the parameter space of our language model to the vicinity of common syntactic patterns. This greatly reduces the number of the effective parameters and enables the model to focus on the details of each author's special syntactic styles.

Also, we use a shared fully-connected layer and a set of independent fully-connected layers to respectively learn the shared syntactic patterns and each author's special syntactic writing styles. A gate unit shared between authors is responsible for deciding whether to use author-specific information at different positions of the texts. The rationale behind this design is that we assume that only a small part of the patterns appeared in the POS sequences comes from author-specific writing styles, while most patterns come from the standard syntax. This structure has two major benefits: (i) the gate can prevent the model from overfitting to each author's rather limited training data since it makes the decision based on the entire training set, and (ii) learning common syntactic patterns with a small set of shared parameters makes the language model more data-efficient and accurate.

To summarize, our main contributions are as follows:

• We propose a novel POS-level language model to effectively encode the topic-agnostic syntactic writing style for each author. As far as we know, we are the first to conduct authorship verification by capturing the authorspecific correlations in the sequences of POS tokens with language models.
• We propose a novel gated architecture to learn the author-specific language models by utilizing a relatively small group of parameters to learn the common syntactic patterns shared across all authors. The model is dataefficient and less prone to overfitting.
• Extensive empirical results on four publicly available datasets demonstrate that our model can effectively learn each author's unique syntactic writing style and achieve significant accuracy improvements compared to other state-of-the-art methods, especially in cross-topic scenarios (over 5% in terms of AUC-ROC).

2 Preliminaries

2.1 Problem Statement

Suppose there is a group of authors A, for each author ai ∈ A we have a small set of texts Si = {si1 , si2 , · · · , sik } that is known to be written by ai. Now there is also a text of unknown author denoted as qt, the purpose of authorship verification is to determine whether qt is written by author ai.

{ai, qt} is called an authorship verification case. Figure 1 shows two toy cases.

2.2 Language Model for Authorship Verification

The core idea of neural language model [Bengio et al., 2003] is learning a neural network to model the conditional probability P(wi|w0, · · · ,wi-1) for each sequence w0, · · · ,wN. w0, · · · ,wi-1 is called context and wi is a token that is either a character or a word. Typically, this model consists of two components: a recurrent neural network unit f synthesize the correlations from the context into a hidden representation hi = f(hi-1, wi-1); and a fully-connected decoding layer g to map the hidden representation to the probability distribution of next token: P(wi|w0, · · · ,wi-1) = g(hi).

To conduct authorship verification, a set of author-specific language models can be learned for each author respectively. Each author's special writing habits will lead to differences in learned conditional probability P(wi|w0, · · · ,wi-1), and a text is expected to have high probability with respect to its sincere author's language model.

3 Methods

Figure 2 gives an overview of the gated POS-level authorspecific language model. First, we convert the raw texts into the sequences of POS tokens. These sequences are then embedded and fed into a vanilla RNN shared by all authors to capture the context information. On top of that, we apply gated decoding to predict the next token in the sequence.

3.1 POS-Level Language Model

Instead of the traditional character-level language model, we propose to use a novel POS-level language model to learn author-specific syntactic writing styles. Specifically, we utilize a POS tagger pre-trained on a large external data set [Matthew and Ines, 2015; Qi et al., 2018] to convert the raw texts into sequences of POS tokens. Then we learn authorspecific language models upon these sequences, that is, predict the next POS token based on previously observed POS sequence. The POS-level language model can effectively capture the syntactic writing styles with limited training data while the character-level language model often fails to do so.

Why Is Syntactic Writing Style Important for Authorship Verification?

The syntactic writing style is important because it is topicagnostic. Consider the two toy examples in Figure 1. The two sentences in the first case convey similar meaning with exactly the same words, but the arrangement of their POS token sequences are completely different. For the second case, the two sentences have different topics but similar POS token sequences. It is evident that the syntactic writing style represented by the arrangement pattern of POS token sequences is more generalizable for authorship verification in cross-topic scenarios.

Why Can POS Level Language Model Better Learn Syntactic Writing Style?

The language model essentially models the joint probability distribution of a sequence of discrete random variables. This distribution has an exponentially sized sample space, thus the language model will need a large amount of training data to explore this huge sample space and accurately fit the distribution. However, in the authorship verification problem, there are typically not enough training data to do this. As a result, the char-level language model is likely to overfit to some special local correlations in the training set.

To address this problem, a potential solution is to leverage external knowledge to constraints the size of the sample space. We can consider that there is a subspace consisting of all the "correct" sentences obeying the common syntactic rules in the sample space. Each author will have a different probability distribution in this subspace according to their syntactic writing style. With this intuition, we leverage the general author-independent syntactic rules obtained from POS taggers trained on external large-scale data sets to limit the sample space of the language model to the vicinity of "correct" sentences, enabling the author-specific language models to focus on each author's subtle syntactic style. From another perspective, the vocabulary size of the POS is only 19, compared to the character (80) and the word (several thousand). This greatly reduces the number of effective parameters of the model while retaining its expressive power. Besides POS tags, we also explored the possibility of using more explicit syntactic information obtained from dependency parsing. However, it performs consistently worse than POS tags.

3.2 Gated Neural Language Model

Given a sequence of POS tokens P = {wt}mt=1 known to be written by author ai ∈ A, we first convert them to embeddings {et}mt=1 with a embedding layer. Then we generate the representation of context {ht}mt=1 utilizing a vanilla RNN unit to capture the correlations in the POS sequence:

The parameters of the embedding layer and the RNN are shared between all authors. We choose vanilla RNN rather than more powerful variance such as LSTM because we find experimentally that LSTM typically provides rather limited performance improvements. We speculate that this might be due to the limited training data which prevents the LSTM to learn meaningful long-range correlations.

Then, the representation of context is fed into a decoder to generate the estimation of the distribution over the next token. This decoder consists of three parts:

At the foot of the

Figure 2: Gated neural language model structure overview.

• (i) A shared fully-connected layer with parameters {Wshare , bshare } to learn the common syntactic writing patterns among all authors:

• (ii) A separate fully-connected layer with parameters {Wai , bai } for each author ai ∈ A to learn special syntactic writing styles:

• (iii) The final output is the linear weighted average of the outputs of the previous two parts performed by a shared gate unit parameterized by {Wgate , bgate }:

Finally, a softmax function is applied to the raw logits output to obtain the author-specific estimated distribution of the next token:

To train the network, we minimize the sum of the crossentropy loss between the estimated distribution and the ground-truth next token on every step t.

Why Does Our Proposed Architecture Learn Author-specific Syntactic Writing Style More Accurately?

The assumption behind this design choice is that there is a large intersection between the writing patterns of the authors, and only a small part of special writing patterns can be utilized to distinguish these authors. With this architecture, we learn the shared writing patterns with a small set of shared parameters, and the author-specific parameters can focus on each author's special writing styles.

Also, note that the parameters of the gate are shared, thus the ratio to include author-specific information at each step of the text sequence is determined by all training data. As a result, it will be less likely for the model to overfit to the training data of each author.

3.3 Verification

After learning a language model for each known author respectively, to conduct authorship verification on a test case {ai, {wt}mt=1 }, we need to calculate the losses z = {zai |ai ∈ A} of this text with respect to the language model of each author respectively:

Then, we normalize these losses:

This normalized loss for ai can be used as a score for classification. The smaller the loss, the more likely this text is written by the author ai.

The rationale behind this choice is that due to the inherent variability of neural language models, it's difficult to directly compare the losses of different texts with respect to the same language model. Instead, we can only compare the losses of the same text with respect to different author's language models.

4 Experiments

In this section, we conduct extensive experiments to evaluate the proposed method. First, we introduce the settings of the experiments, including the datasets used for evaluation, the baseline methods to compare, the implementation details of our method, and the metrics. Then we compare the performance of our proposed method with other state-of-the-art authorship verification methods. Finally, we conduct ablation studies to evaluate the contribution of each component of our model.

4.1 Data Sets

As shown in the Table 1, we used 4 publicly available authorship verification datasets, which were widely used by previous studies [Halvani et al., 2017; Bevendorff et al., 2019a; Bagnall, 2015; Halvani et al., 2018], with different genre and sizes. Each case in these datasets consists of exactly one document of unknown authorship and at least one document from a known author. PAN-15 comes from a well-known authorship verification competition [Stamatatos et al., 2015]. Recently, the authors of PAN-15 conduct a detailed analysis of the dataset and find several deficits such as special characters introduced during the processing of the dataset and accidental text overlap. Based on these insights, they constructed an improved authorship verification corpus named Gutenberg [Bevendorff et al., 2019a]. This dataset eliminates the aforementioned bias and flaws, enabling the dataset to evaluate the performance of authorship verification methods more fairly. Therefore, this dataset is considered the most important one in our experiments. Nevertheless, we still included the PAN15 dataset for completeness. The Enron data set comes from the public Enron emails dataset. The samples are manually selected and cleaned to make sure of the quality [Halvani et al., 2018]. The Reddit dataset is obtained from [Halvani et al., 2017].

These data sets are respectively divided into two parts, the development set, and the test set. The PAN-15 and Gutenberg datasets have been pre-divided by the provider. The other two datasets are randomly divided according to the ratio of 1: 4 following the common practice in the research of authorship verification [Halvani et al., 2017].

4.2 CompetingMethods

We compare our model with five other methods, including two baseline methods, BAFF [Bevendorff et al., 2019a] and its variation that use POS tags as inputs, two non-deep learning methods GLAD [H¨urlimann et al., 2015] and Compression [Halvani et al., 2017], and a deep learning based method MultiHead [Bagnall, 2015], which won the 1st place on the PAN 2015 author identification task [Stamatatos et al., 2015]. We chose these methods because they are the state of the art in the field of author verification.

We reimplement the two baseline methods with Python, and MultiHead with PyTorch [Paszke et al., 2019]. For other methods, we use codes obtained from their authors.

4.3 Implementation Details

We implement the proposed method with PyTorch and run all the experiments on a GPU with 11GB memory.

How to perform POS tagging is not the focus of this article, therefore we rather arbitrarily choose the popular open-source POS tagger Spacy [Matthew and Ines, 2015]. Other POS taggers such as StanfordNLP [Qi et al., 2018] can achieve similar results.

In order to conduct fairly performance comparison to make sure the performance improvement of our method comes from the POS level language model and gated decoder, we use the vanilla RNN same as MultiHead.

The most important hyperparameters of our model are the size of the embedding layer and the size of the hidden layer. These hyperparameters can be effectively selected on the development set and do not vary much on different datasets. Except for the Enron dataset where the best hidden size is 16, on other datasets the model achieves the best results on the development set when the hidden size is 64.

In training, we follow the common practice to calculate the gradient with truncated backpropagation through time [Werbos and others, 1990; Graves, 2013]. Specifically, we truncated the gradient for 20 time-steps and perform gradient descent utilizing vanilla SGD optimizer combined with gradient clipping at 0.25. All models are trained for 100 epochs and the learning rate is set to 5. With these settings, we achieve stable performance on the development set.

4.4 Performance EvaluationMethods

We use two complementary indicators: AUC-ROC and AUCPR to measure and compare model performance. The AUCROC is the area under the ROC curve, summing up true positives against false positives. The AUC-PR is the area under the curve of precision against the recall. We use average precision in [Manning et al., 2008], a widely-used method, to calculate AUC-PR. We chose these two metrics because they are classification-threshold-invariant metrics. It allows us to focus on comparing the predictive power of each method. For a specific application, a threshold can be chosen to trade between precision and recall.

Due to the inherent variation of the algorithm, the experimental results reported in this paper comes from the average of 10 independently running with the same setting.

4.5 Performance Comparision

This section examines the performance of our method on aforementioned datasets. There are two major questions we aim to address:

• Is it necessary to utilize deep learning methods for authorship verification problem?
• Is our method more accurate than the state-of-the-art deep learning based method?

The AUC-ROC and AUC-PR performance of our method and other competing methods are listed in Table 2. It is evident that deep learning based methods MultiHead and our GatedLM outperform traditional models on all four datasets, which indicates that deep learning based methods can beat sophisticated manually designed features. Also, notice that the BAFF on POS perform worse than the raw BAFF on character. This implies that simply using distribution of POS tags is unable to unleash its full potential.

Comparing our method with MultiHead, we can obtain the main result that the POS level information and the gated neural language model enable our method to achieve significant performance improvement. In particular, our method achieves the most significant performance improvement in cross-topic scenario on Gutenberg dataset.

4.6 Ablation Study

In this section, we perform ablation studies on the two key components of our proposed method to examine their contribution to performance improvement respectively. We aim to answer the following three key questions:

• Is it necessary to share parameters between language models of authors?
• Does our gated neural language model achieve performance improvement over the multi-head structure?

• Does POS information have advantages over char information, and under what circumstances?

To answer the first question, we build a baseline model, which learns a completely independent language for each author. This model has no shared parameter.

From the experimental results listed in Table 3 we can find that both POS-level information and gated architecture make major contributions to the performance improvement. The baseline model without any parameter sharing performs significantly worse than multi-head and gated architecture, which indicates that it has difficulties in learning accurate language model for each author with extremely limited training data. Therefore, a well-designed cross-author parameter sharing is the key to learn more accurate author-specific language models by reducing the number of effective parameters.

Also, our gated architecture achieves higher accuracy compared to the multi-head when both trained on POS level inputs. This demonstrates that our gated decoder can indeed distinguish the author-specific writing styles from common patterns and learn the common patterns with shared parameters. As a result, our model is less prone to overfitting and can learn a more accurate language model.

Not surprisingly, when using the same model architecture, the POS-level language model outperforms the char-level language model by a large margin on the Gutenberg dataset but only achieves moderate performance improvement on the PAN-15 dataset. This is due to that the PAN-15 dataset contains special characters introduced during the process of the dataset. Character level models can easily overfit these special characters and obtain biased high accuracy.

In order to quantitatively compare the importance of the author-specific decoder and shared decoder, we calculate the mean absolute logits of them respectively. We also calculate the mean value of the gates. The experimental results listed in Table 4 demonstrate that the response amplitude of the shared decoder is much larger than the amplitude of the authorspecific decoder on all datasets, which provide evidence to further support our claim that the model can learn common writing patterns by shared decoder and author-specific styles by stand-alone decoder for each author.

5 RelatedWork

The authorship verification problem is first proposed and studied in [Koppel and Schler, 2004], and is closely related to authorship attribution problem. While being almost solved on long texts [Bevendorff et al., 2019a], authorship verification is still a challenging task on short texts.

5.1 Traditional Authorship Verification

There are numerous authorship verification methods attempting to tackle this problem from different aspects [Stamatatos, 2009].

The very first work on authorship verification [Koppel and Schler, 2004] proposed an unmasking method that mainly focused on the analysis of most frequent function words. Writeprints [Abbasi and Chen, 2008] extract more than twenty stylometric features including lexical, syntactic, and structural text features to conduct authorship analysis. BAFF

[Bevendorff et al., 2019a] relies on several simple measures (e.g, TF-IDF) defined upon char level trigram. The distribution of POS tags can also be utilized to measure the style differences [Zhao et al., 2006]. Compression is another wellstudied path to authorship verification which is first proposed in [Khmelev and Teahan, 2003] and recently improved by [Halvani et al., 2017].

5.2 Deep Authorship Analysis

Only a few studies have been done on deep authorship verification and deep authorship attribution. In [Bagnall, 2015], a character level language model is learned for each author to capture their unique writing styles. Texts will have a higher probability coming from the language model of its true author. Recently, several methods use convolution neural network to identify the authorship. Their main differences are the formats of the inputs. [Hitschler et al., 2017] first transforms the raw texts into sequences of POS tags, then use a convolution neural network to predict the author. [Shrestha et al., 2017] proposes to use character n-gram as the input of the convolution neural network. [Ruder et al., 2016] proposes to combine the char-level input and the word-level input with a multi-channel neural network.

6 Conclusions

We have presented our novel POS-level (Part of Speech) gated RNN based language model that can learn the authorspecific syntactic styles with high data efficiency. We first propose to use a POS-level language model to obtain the syntactic level information. Then we design a novel gated decoder architecture to improve parameter sharing and enable data-efficient learning of the author-specific language model. Extensive experimental results show that our model outperforms other state-of-the-art competing methods by a large margin in terms of accuracy.