Joint Representation Learning of Legislator and Legislation for Roll Call Prediction

1 Introduction

Quantitative political science aims to understand how the government behaves via analyzing legislative data. Roll call data, historical records of legislators' votes on a set of issues, attracts intensive attention from quantitative political scientists because it presents actions of law-makers. Researchers have analyzed roll call data from the United States Congress [Clinton et al., 2004] or the British Parliament [Cowley and Garry, 1998] to reveal the political leaning of congressmen. An example of a legislative record can be seen in Figure 1. Each record contains voting results and information about the legislation with its title and description. A legislation is initialized by a group of congressmen, consisting of a sponsor and several co-sponsors. Voting results made by legislators include Ayes (yes), Noes (no), and Not Vote.

The most popular approach for roll call data modeling is named ideal point model [Clinton et al., 2004]. It represents legislators and legislation as points in a latent space. And the voting behavior of a legislator can be characterized as the distance between his/her position and the target legislation in the space. With this setting, researchers explore to predict the voting results of legislators. However, the positions of legislators are learned based on their historical voting results, ignoring other context information. This results in poor predicting accuracy. In practice, the voting behavior of the congressman is affected by multiple factors, the topic of legislation, the influence of other congressmen, personal preferences, etc. Although some extensions [Gerrish and Blei, 2011] are made to incorporate textual information for better modeling the legislation, the ideal point model still lacks the power to capture a richer context of the legislative record.

∗ Indicates the two authors have equal contributions.

†Contact author

To tackle this problem, we propose to learn representations for both legislators and legislation incorporating context information of a legislative record for better prediction of roll call results. For legislators, we introduce relationships among legislators and utilize graph convolutional networks (GCNs) for representation learning. For legislation, we encode its narrative description via recurrent neural networks (RNNs) for the representation learning. Following the ideal point model, we map representations of the legislation and legislators into the same embedding space. A triplet loss is utilized as the objective function to learn the two kinds of representations jointly. Therefore, legislators' degree of approval to a target legislation can be measured by the Euclidean distance between their representations and the legislation's representation. We collect a dataset from the Congress website in the U.S. between 1993 and 2018 for the evaluation. The major contributions of this paper can be three-fold.

• We represent a congressman using their background information and jointly learn dense representations of all congress members incorporating their network information via graph neural networks.
• We align representations of legislation and legislators in the same vector space and introduce a triplet loss for the joint training.
• We collect a dataset including legislative records of U.S. Congress. Experimental results from this self-constructed dataset show the effectiveness of our framework. To the best of our knowledge, this is the first dataset constructed for voting result prediction in the political area. We will make it public.

2 Dataset Construction

The dataset is collected from the Congress website in U.S. 1 , including both legislation information and roll call results from 1993 to 2018. There are 215,857 legislation in total, with 2,234,082 voting records. 2,347 legislators are involved in this dataset. We present the change of numbers of legislation along the time span in Figure 2a.

Legislator. A legislator is a person who writes and passes laws, especially someone who is a member of a legislative body. Our targets here are Members of the U.S. House of Representatives. The total number of voting representatives is fixed by law at 435, and the term of office of the members is two years, with no limit for re-election (the re-election rate is around 81.91%). Every member owns an ID and has information about party and state.

Legislation. The legislation is a law which has been promulgated by a legislature or in the process of making it. Before an item of legislation becomes a law, it may be known as a bill and can be broadly referred to as "legislation" (we use legislation through this paper). Every legislation belongs to a specific policy area and it has related textual information of title and description. We list the proportion of legislation in different policy areas in Figure 2b. A legislation is initialized by a group of congressmen, consisting of a sponsor and several co-sponsors. The distribution of the numbers of co-sponsors is shown in Figure 2c.

Roll call record. A roll call record contains voting results of legislators with respect to a legislation. There are three types of voting results, namely, "yeas" (yes), "nays" (no), and "not vote". The distribution of voting results can be seen in Figure 2d.

1 www.congress.gov. The dataset is avaliable on http://www.sdspeople.fudan.edu.cn/zywei/data/fudanUSRollCall.zip and the codes for this paper is avaliable on https://github.com/lxqjdai/Joint-Representation-Learning-ofLegislator-and-Legislation-for-Roll-Call-Prediction

3 Task Formulation and Overall Architecture

Given a legislation and a set of legislators, we aim to predict the voting result for each legislator. We describe the prediction task starting with the definition of symbols.

• A set of congress members M = {m1,m2, . . . ,mk} and every member mi has an ID: mi(ID), information of party: mi(p) and state: mi(s).
• A set of legislation L = {l1, l2, . . . , ln} and every legislation li has a description li(d), and a sponsor co-sponsor network li(s).
• A set of roll call voting records R = {r(mi, lj)|1 ≤ i ≤ k, 1 ≤ j ≤ n}, r(mi, lj) stands for the voting result of legislator mi to legislation lj and it has three labels, namely yea, nay and not vote.

The overall inference workflow can be seen in Figure 3. It mainly includes three components, namely, legislation representation learning, legislator representation learning, and voting result prediction. We learn the representation of legislators with graph convolutional networks utilizing sponsor co-sponsor network and learn the representation of legislation using recurrent neural network based on its description. The voting preference of a legislator with respect to a legislation is measured by the similarity between their representations. We then rank legislators in terms of their preferences (higher preference score means higher tendency to vote yes) and predict their voting results according to a provided voting ratio (proportion of different voting results). The voting ratio of a legislation is automatically determined by a semantic GCN model.

For the training, we introduce a triplet loss as the objective function to jointly learn of the two kinds of representations. We will introduce our joint training framework in Section 4 and the ratio aware result prediction module in Section 5.

4 Joint Representation Learning of Legislation and Legislators

We explore to learn dense vectors to include complex information for both legislation and legislators. And then, we aim to align both kinds of representations within the same vector space via joint training. The joint learning framework mainly includes three components, namely, GCN-based legislator representation learning, LSTM-based legislation representation learning, and joint training based on triplet loss.

4.1 GCN-based Legislator Representation Learning

We initialize the representation of congressmen by their background information and then update their representations via GCNs taking legislator network into consideration.

Representation initialization. We first map every congress member to a continuous low-dimension vector, Xlgt, with three components, namely, ID, party, and state.

where XID stands for ID which is unique for every Congress member, XParty denotes democracy or republican and XState stands for the state where the member is elected.

Legislator network construction. Each legislation is initialized by a sponsor and several co-sponsors. It is reasonable that the relationship among legislators can be reflected from such political activity, i.e., raise a legislation together. In specific, we regard legislators as nodes and use the connections of sponsor and co-sponsor as the edge in the network. We consider all the sponsor and co-sponsor relationships in a period of time to obtain the adjacency matrix A. The element aij means the number of legislation that mi and mj co-sponsor.

GCN-based legislator representation updating. GCNs [Kipf and Welling, 2016] are neural networks operating on graphs and including features of nodes based on properties from their neighborhoods. In our scenario, we utilize GCNs for the representation updating of legislators based on the sponsor co-sponsor network. Each layer of our GCNs is structured as follows:

Here, A is the normalized adjacency matrix of the network. W (0) ∈ RC×H is an input-to hidden weight matrix for a hidden layer. W (1) ∈ RH×F is a hidden-to-output weight matrix. Here we take Xlgt of all legislators as input and update their representation accordingly.

4.2 LSTM-based Legislation Representation

As for the legislation, we use the title and description for its representation learning. We use the recurrent neural network (LSTM [Hochreiter and Schmidhuber, 1997] in our case) to encode the text information of the legislation as the legislation feature(Xlgn).

li(d) stands for the text information of the legislation lj.

4.3 Joint Training via Triplet Loss

For inference, the distance between legislators and legislation should represent the legislators' preference for voting for the legislation. We, therefore, want to align representations of both the legislation and legislators within the same space. In order to achieve this, we introduce triplet loss [Chechik et al., 2010; Wang et al., 2014] for the joint training.

At each training iteration, we sample a mini-batch of triplets. Each one consists of the legislation and a pair of legislators, one is positive mj + and the other is negative mk -. Voting results of the two legislators should satisfy r(mk -, li) < r(mj +, li) with the rule of YES < Not V ote < No. The goal of triplet loss is to push away the representation of negative legislator Xlgt(k)- from the representation of the legislation Xlgn by a distance margin ϵ > 0 compared to the representation of positive legislators Xlgt(j)+. Here is the form of hinge loss to implement this:

5 Ratio Aware Voting Result Prediction

Based on representations of legislators and the target legislation, we measure the preference of legislators to vote for this legislation by computing their Euclidean distances in the embedding space. Legislators are then ranked by the distance in ascending order (the closer the distance is, the higher tendency to vote for this legislation). With a provided voting ratio, i.e., the proportion of different voting results, we are able to classify legislators as supporters/neutrals/opposers in turn. In practice, we predict the ratio of voting results for the legislation automatically. We explore to utilize a GCN-based model for voting ratio prediction, named semantic-GCN.

Legislation network construction. We treat each legislation as a node and the similarity between two legislation as the weight of the edge for legislation network construction. Following [Liu et al., 2019], we extract keywords for each legislation using the Text-rank algorithm and use the average of keywords' glove vectors as its representation. Based on the representation, we compute the similarity between legislation.

Semantic-GCN based voting ratio prediction. Take the representation of legislation learned in Section 4.2 as initial input, we utilize GCN to update the representation of legislation. We concatenate the GCN output with the initial representation of the legislation and predict the voting ratio through a two-layer MLP regression. Note that the output is a probability distribution with three entries corresponding to three voting results.

6 Experiment

6.1 Experiment Setup

Experimental dataset construction. Considering legislation in the dataset spread in a large range of time span, we set a window size to extract roll-call data to build a number of experimental datasets for the evaluation of our model. In specific, we continuously extract five years of data from 1993 to 2018 each time to construct a single experiment set, and this results in 22 splits with overlaps. In each split, we use data in the first four years as the training set and the data in the last year as the testing set. Moreover, we randomly select 20% of legislation in the training set for parameter tuning. For instance, the first experiment set is constructed based on legislation raised from 1993 to 1997 (training on the year 1993 to 1996 and test on the year 1997). We calculate accuracy for each experiment set and report the average accuracy of the 22 sets for evaluation.

Legislator network construction. The information of sponsor and co-sponsor network might uncover future information for roll call result prediction of legislation in the same period of time. Therefore, We use sponsor and co-sponsor information in a period of time to construct the legislator network for the representation learning of legislators. In order to avoid the issue of future information leaking, we use the legislator network constructed in the training set for the inference of legislation in the test set.

Implementation details. The GCN for legislator representation learning is two-layer and the size of the hidden layer is 32. The dimension size of the legislator representation is 32 (16 for ID, 8 for the party, 8 for state). The initial learning rate is 1 × 10-4 . We apply early stop to keep the model from over-fitting. The number of the hidden layer in the Semantic GCN is 32. The parameters are tuned on the validation set.

6.2 Models for Comparison

We compare our model with two state-of-the-art approaches.

clinton [Clinton et al., 2004] This model utilizes the Bayesian model to compute positions of legislators and legislation in the ideal point space. Distance between representations of legislators and legislation is used to characterize the voting behavior.

gerrish [Gerrish and Blei, 2011] This model extends the ideal point model with textual information of the legislation and embeds the positions of legislation using text regression.

We employ different algorithms for three components in our framework as ablation study, namely, legislation representation learning, legislator representation learning and voting result prediction. Here are the details of comparative models.

LSTM+Deepwalk This model uses deepwalk [Perozzi et al., 2014] to obtain the representation of legislators. LSTM is used for legislation representation learning. It concatenates the two representations and uses a two-layer perceptron model for voting result prediction.

LSTM+Node2vec This model is similar to LSTM+Deepwalk except it uses the node2vec [Grover and Leskovec, 2016] for legislator representation learning.

LSTM+GCN LSTM, GCN and a two-layer perceptron are used for legislation representation learning, legislator representation learning and voting result prediction.

LSTM+Party This model is similar to LSTM+GCN, except we use only party information to represent legislators.

LSTM+GCN+triplet loss This is our model that utilizes the triplet loss to update the representations of legislation and legislators jointly. The voting ratio of the legislation is obtained via semantic GCN.

LSTM+GCN+triplet loss (GT) We use the ground-truth voting ratio in our model and report the results for reference.

6.3 Overall Performance

The overall performance of different models are shown in Table 1. We have following findings:

• The performance of clinton is worse than all other methods, which shows the limited prediction power of the ideal point model that ignores context information of legislative records. By incorporating the textual information, gerrish can improve the performance by a large margin.
• By using dense representations for legislation and legislators, the prediction models achieve much better accuracy compared to the classic model clinton.

• Compared LSTM+GCN with LSTM+deepwalk and LSTM+node2vec, we can see the effectiveness of using graph neural network to model relationships among legislators for their representation learning. It is interesting to note that the performance of LSTM+party that uses party information to stand for a legislator is quite promising. We will further analyze this in the next sub-section.
• Our model LSTM+GCN+triplet loss performs better than all other baselines. This indicates the effectiveness of using triplet loss for joint training the representation of legislation and legislators.
• By using the ground-truth voting ratio, the model can further improve the performance compared to models with automatic ratio prediction results.

6.4 Performance of Voting Ratio Prediction

We further analyze the performance of our semantic-GCN for automatic voting ratio prediction. We compare four different models with various configurations for legislation representation learning and adjacency matrix construction.

S-GCN-Cos-Glove We learn the initial representation of legislation via averaging the embedding of words it contains. The similarity between two legislation is computed cosines similarity of the two representation vectors. A two-layer GCN is then used to update the representation of legislation for voting ratio prediction.

S-GCN-Cos-LSTM This model is the same as S-GCN-CosGlove, except that it uses LSTM to initialize the representation of legislation.

S-GCN-Cos-Cat-LSTM On top of S-GCN-Cos-LSTM, this model concatenates the output of semantic GCN and LSTM as the representation of a legislation for the final prediction. This is the one we used for voting ratio prediction.

S-GCN-Coo-Cat-LSTM This model is the same as S-GCNCos-Cat-LSTM, except the adjacency matrix for GCN is built via co-occurrence of keywords.

We use MSE as the evaluation metric. The lower the score is, the better the model is. The results can be seen in Table 2. The performance of S-GCN-Cos-LSTM and SGCN-Cos-Glove shows no significant difference. However, S-GCN-Cos-LSTM's performance is relatively more stable across different years. So we choose LSTM to continue. The performance of S-GCN-Cos-Cat-LSTM is better than that of S-GCN-Cos-LSTM, which shows that the concatenation works. The performance of S-GCN-Cos-Cat-LSTM is apparently better than that of S-GCN-Coo-Cat-LSTM, which shows that the cosine similarity works better than co-occurrence when constructing the adjacency matrix.

6.5 Influence of Party Stand on Voting Behavior of Legislators

Legislators in the same party tend to have the same stand to a specific legislation. Therefore, party is a vital feature for voting result prediction. The performance of LSTM+party in Table 1 confirms this. In this sub-section, we look deep into the relationship between the voting behavior of legislators and their party stand.

Intra-party consistency. We first define the main stand of a party with respect to a specific legislation as the majority voting result of party members. We then measure the intraparty consistency as the percentage of legislators in the party who make the same voting decision as the main stand. Averaging on all the experimental sets for both parties, we get that the intra-party consistency is 88.42% (Democratic: 87.05% VS Republican: 89.11%). This shows that there are a number of legislators who do not follow the main stand of their party.

Deviation degree of legislator. We say that a legislator deviates from the party if he/she did not follow the main stand of the party in a voting process. We compute the deviation degree DRmi of a legislator mi, with the following formula: DRmi = Dmi Tmi . Dmi is the total number of deviations of mi, Tmi is the total number of voting records of mi. The distribution of deviation degree of legislators is shown in Figure 4a.

Behavior prediction for legislators of high deviation degree. We select the top 5% of legislators with higher deviation degree. Their behaviors are supposed to be more difficult to predict. The results of the prediction accuracy for these legislators are shown in Table 3. Experimental results show that our model is more effective for modeling behaviors of these legislators with higher deviation degree.

Deviation degree of legislation. We further study the degree of deviation for legislation to see in what policy areas legislators tend to vote differently with the main stand of their party. We define the deviation degree DRlj of legislation lj with the formula: DRlj = Dlj Tlj . Dlj is the number of legislators who deviates from their party in voting this legislation and Tlj is the total number of legislators involved in this voting. We then define the deviation degree of policy area Pk as the average deviation degree of legislation it includes:

DRPk = ∑lj∈Pk Dlj #lj∈Pk . The distribution of deviation degrees for different policy areas is shown in Figure 4b. We can infer that policy areas related to animals, food, and sports, which are highly related to the individual personalities instead of party interests, obtain a higher deviation ratio.

6.6 Case Study

To show the efficiency of our framework qualitatively, we select a specific legislation and get the representation of the legislation and legislators. Then, we project representations into the 2-D space through PCA. The visualization can be seen in Figure 5. We can see that legislators who vote for this legislation stay close to the representation of this legislation and those who against the legislation locate far away. This shows that the triplet loss efficiently encodes the roll call result information into the representation of legislators and legislation.

7 RelatedWork

There are two related research fields, namely, the application of Graph Neural Network and roll call result prediction.

Application of GraphNeural Network. The most popular task of Graph Neural Network explores to learn dense representation for nodes in the network. Traditional GNN models usually use explicit graph Laplacian regularization [Zhu et al., 2003; Belkin et al., 2006; Weston et al., 2008] while current research borrows the idea from word embedding learning [Mikolov et al., 2013]. Approaches in the latter category includes DeepWalk [Perozzi et al., 2014], LINE [Tang et al., 2015]) and node2vec [Grover and Leskovec, 2016]. Other graph embedding methods incorporate text information [Tu et al., 2017; Sun et al., 2016]. More recent literature, i.e., GCNs [Kipf and Welling, 2016], makes progress in combining graph structure with supervised machine learning targets directly. Based on GCNs, researchers explore different applications including semantic role labeling [Marcheggiani and Titov, 2017], named entity recognition[Cetoli et al., 2017] and structured label classification[Chen et al., 2017].

Roll call result prediction. Modeling the policy preferences of legislators toward legislative outcomes is the main research problem of modern legislative behavior research. Based on legislative voting records, researchers make inferences about the policy preferences of legislators [Clinton et al., 2004], the conflicted political issues [Cowley and Garry, 1998], the cohesiveness of parties [Rosenthal and Voeten, 2004] and the existence of intra-party factions [Rosenthal and Voeten, 2004]. Most of transitional research is based on ideal point model [Clinton et al., 2004; Jackman and Rivers, 2004]. [Gerrish and Blei, 2012] develops the issue-adjusted ideal point model. And there are also some game-theoretic models of congressional vote prediction [Ghoshal and Honorio, 2016; Irfan and Gordon, 2018].

8 Conclusions and FutureWork

We focus on the task of roll call results prediction based on legislative data. We build up the first dataset for this task and propose a framework to learn dense vectors for both the legislation and legislators to make the prediction of roll call result. Experimental results show the effectiveness of our framework. In future work, we are interested in conducting graph convolutions dynamically to help draw the picture of the change in the Congress members.

Acknowledgments

This work is partially supported by National Ministry of Science and Technology of China (No. 2018YFC0830600), National Natural Science Foundation of China (No. 61751201, No. 71991471) and Science and Technology Commission of Shanghai Municipality Grant (No. 17JC1420200, No.18DZ1201000).