Generating Reasonable Legal Text through the Combination of Language Modeling and Question Answering

1 Introduction

The improvement of Language Modeling [Mikolov et al., 2010; Vaswani et al., 2017; Radford et al., 2018; Devlin et al., 2019] has greatly changed the landscape of NLP, and begin to shed lights on automatic text generation, which can reduce the human workload on writing documents. The successful examples are the smart reply [Kannan et al., 2016] and the smart compose in Gmail, helping people to type the fixed daily utterances in e-mail editor.

The similar efforts have being made in the legal domain [Alschner and Skougarevskiy, 2017] to automatically generate the official legal documents, even since 1990s [Branting, 1998]. However, the generating of legal text faces greater challenges than ordinary texts due to its own characteristics. There are at least two requirements for the generation of legal documents: (1) the syntax should be correct, and (2) the logic should be reasonable. The syntax correctness problem has been mainly relieved by the current pretrained-then-fine-tuned language models such as GPT [Radford et al., 2018]. But the current language models still lack the ability to keep logical rationality [Mao et al., 2019; Guan et al., 2020]. For example, the text may be generated by a language model such as "Someone misappropriated 100,000 yuan of public funds. In accordance with Article 100 of the Criminal Law of the People's Republic of China 1 , the judgment is as follows ...", which is syntax-correct. But the language model here does not know what Article 100 is, even it has been trained on a lot of judicial documents. In fact, Article 100 in the Criminal Law of the People's Republic of China is about the crime reporting system, which logically conflicts with the fact of the misappropriation of public funds.

∗Contribution during internship at Ping An Technology.

In another hand, the template-based generation seems a possible solution, but it needs large manual efforts for creating templates [Branting, 1998], and cannot be generalized to a more complex scenario. Meanwhile, a method for generating variational templates is proposed [Ye et al., 2020] for further generating text from tables, but can not be easily extended to the legal domain. However, the template-based method meets another challenge that requires the knowledge of implementing the templates to avoid logic conflicts.

To solve this problem, we introduce the Knowledge Base enhanced Question Answering technique to the existing language modeling in order to generate the logical part. In our pilot study, when generating the "Article 100", the language model was very uncertain whether it was correct. In general, where the language model is uncertain, you can ask the system: what applicable law article should be applied to a certain person who misappropriates 100,000 yuan of public funds? The Question Answering component can look up the knowledge base and historical precedents, knowing that it is Article 185 and Article 272 in the Criminal Law of the People's Republic of China, which is further discussed in Subsection 4.3 and Figure 3. We use the slots to represent the logical part need to be addressed in a QA system. And the slots can be generated as special tokens in a text sequence by a trained language model.

1 The English version of the Criminal Law of the People's Republic of China can be found on an official webpage https://www. fmprc.gov.cn/ce/cgvienna/eng/dbtyw/jdwt/crimelaw/t209043.htm.

To implement this idea, we propose a novel method CoLMQA, which (1) combines Language Modeling and Question Answering, (2) generates texts with slots by Language Modeling, and (3) fills the slots by our proposed Question Answering method named Transformer-based Key-Value Memory Networks.

To sum up, our contribution is mainly in four aspects. (1) We convert the problem of keeping logic in the legal text generation into two sub-tasks of generating texts with slots and filling slots with the logical coherent values. (2) With the guiding of this divide-and-conquer idea, we propose a method CoLMQA, which benefits from the Language Modeling and Question Answering simultaneously. (3) We propose a Transformer-based Key-Value Memory Networks, which can encode a long query. (4) The experiment verifies that CoLMQA can generate fluent sentences with slots, and can fill in the correct values to keep the overall logical coherent.

2 RelatedWorks

On methodology, there are two orthogonal lines of research related to our work: language modeling and question answering. And the related works also include the NLP techniques applied to legal documents.

Languagemodeling. The languagemodels are used to predict the words when given context, describing the statistical pattern in a sequence, such as the n-gram in traditional statistical linguistics [Bellegarda, 2004]. With the development of deep learning, neural language models like LSTM [Hochreiter and Schmidhuber, 1997] and RNNLM [Mikolov et al., 2010] enhance the prediction power. Furthermore, the Transformer-based neural language models [Vaswani et al., 2017; Radford et al., 2018; Devlin et al., 2019] have greatly changed the landscape of NLP.

Both traditional statistical language models and neural language models are trained on text corpora to memorize patterns. Although Petroni [2019] and Bouraoui [2020] point out that the masked language model BERT [Devlin et al., 2019] can learn certain types of factual knowledge from large text corpora, BERT cannot be used on the text generation task directly. On the other hand, the autoregressive language model GPT [Radford et al., 2018], GPT-2 [Radford et al., 2019] can generate syntax-fluent text, and see successful applications.

But GPT(-2) faces the problem of unabeling to generate factual aware text [Logan et al., 2019; Mao et al., 2019; Guan et al., 2020]. To generate reasonable stories, Mao [2019] and Guan [2020] both independently conducts the GPT-2's multitask fine-tuning on external common sense datasets (e.g., ConceptNet) to promote GPT-2's awareness of facts. Unlike generating reasonable stories, our task places additional emphasis on the preciseness of law article numbers to keep logic in the legal texts. Just like cardinal numbers in natural language need to be processed additionaly [Andor et al., 2019], so are ordinal numbers in legal documents.

Question answering. Question Answering is a big family of NLP tasks, including QA on Knowledge Base [Berant et al., 2013], machine reading comprehension [Rajpurkar et al., 2016], cloze-style QA [Das et al., 2017], etc. In a nutshell, it provides answers to natural language questions. Our task of filling correct values in generated legal text slots is equivalent to the cloze-style question answering.

Adopting an external memory module, the memory networks [Weston et al., 2015; Sukhbaatar et al., 2015; Bordes et al., 2015] perform well on QA for mainly two reasons: (1) the addressing and reading on external memories help the multi-hop reasoning; (2) the memory module can visit an external knowledge base and scale up to complex reasoning. Key-value memory networks [Miller et al., 2016] extends MemNNs by separating keys and values in memory module, making it suitable for reading the (key, value) style external knowledge [Das et al., 2017; Xu et al., 2019]. The aforementioned memory networks exploit bag of words or RNN to encode queries, keys, and values but cannot encode very long texts well in our slots filling scenario. The closest work to ours is Generative Transformer Memory Network [Dinan et al., 2019], which employs Transformer as encoder and decoder in multi-turn dialogue. In our work, the answer should be selected rather than generated to ensure preciseness.

Legal documents. It's a big human workload to process massive legal texts. So employing NLP methods in legal domain has attracted a lot of attention recently, for instance, charge prediction [Luo et al., 2017; Hu et al., 2018; Chen et al., 2019], question answering [Zhong et al., 2020], and applicable law articles prediction [Zhong et al., 2018]. And the generation of legal texts also draws interests in the NLP community. Alschner [2017] proposed a modified RNN and applied on bilateral investment treaties. Ye [2018] provided a seq2seq method to generate court views (written explanation from judges) from criminal facts.

The legal documents can be roughly categorized as follows: legislative, executive, judicial documents, and contracts [Gostoji´c and Markovi´c, 2019]. The aforementioned methods mainly focus on only one type of documents. However, the contents of the law articles are in the legislative documents, while the numbers of the law articles are mentioned in the judicial documents. Therefore, combining them helps to generate reasonable legal documents.

3 ProblemDefinition

We consider the problem of generating a reasonable legal document d when given a prompt π and a legal knowledge base K. More specifically, we narrow down the type of legal document d to judicial document, so the prompt π is a text sequence for describing the meta information of the judicial document, such as the criminal fact. A legal knowledge base is formalized as K = {(ki, vi)}|K| i=1 , where ki is the ith law article and vi is the corresponding content. Then the task of generating reasonable legal text is to find out the best document d under the constraint between π and K, shown as Equation (1).

It's infeasible to get the global optimal solution in Equation (1) because of the huge search space O(|V|N ), where |V| is the vocabulary size and N is the upper bound of document length. To get approximation solutions, we decompose the original problem with the chain rule as follows: p(d′ |π, K) = ∏ni=1 p(si|π,K, s1:i-1), where d′ = s1:n and si is the i-th sentence in document d′ . We conduct the greedy search on the decomposed parts to get the final document, which means searching a feasible solution for sentence si when given the previous ones s1:i-1 at each step i. With the approximation, the original problem is converted to the following one.

Although the original optimization problem is relaxed, the new one is still challenging because the sentence si relates to the prompt π, previous sentences s1:i-1 and a legal knowledge base K simultaneously. We will show how to tackle the problem proposed in Equation (2) through the combination of Language Modeling and Question Answering in Section 4.

4 Method

In this section we discuss the main components of our proposed method CoLMQA.

4.1 The Overall Architecture of CoLMQA

Before diving into the details of CoLMQA, recall the "Keep Calm and Carry On" meme, which's a typical phrase often being imitated, by keeping the slots but replacing values with others, such as "Keep Calm and Never Give Up". Our method CoLMQA follows a similar style: generate texts with slots and fill them on. In this example, "Carry On" and "Never Give Up" are two different values of the slots in the meme.

More specifically, we divide the challenging task defined in Section 3 into two sub-tasks: (a) text generation with slots and (b) automatic filling slots based on the text context and knowledge base. Therefore, the optimization target in Equation (2) can be further decomposed as Equation (3), where s(-) i is the sentence by keeping slots with placeholder, and v(+) i are the values of slots in sentence s(-) i need to be filled in. Then the optimization of Equation (2) is divided into two sub-tasks of optimization.

Illustrated in Figure 1, CoLMQA contains three parts: the controller, the language modeling, and the question answering. The controller runs the text generation with the following rules, and get the two sub-tasks in Equation (3) optimized separately.

Rule 1 (LM). If there's no slot in existing text, call LM to generate next sentence.

Rule 2 (QA). If there are slots in existing text, call QA to fill slots.

Rule 3 (End). If there's an END symbol in existing text, finish the generation.

4.2 Generating Texts with Slots by Language Modeling

As shown in Figure 2, there are two phases in using a language model to generate texts with slots. The first phase is fine-tuning. In this phase, after replacing the corresponding values in the original text with slots, we put the legal documents into a pre-trained model for fine-tuning, where different slots occupy different positions in the vocabulary.

si 2 Vword [ Vslots

Figure 2: Generation of texts with slots.

The second phase is to use the fine-tuned language model to generate texts with slots. We choose GPT as the language model in our work. So Vwords is the original vocabulary used by the pre-trained GPT, and Vslots includes the additional placeholders in the fine-tuned GPT.

4.3 Filling Slots by Transformer-based Key-value Memory Networks

Briefly, our proposed method is in a family of Key-Value Memory Networks [Miller et al., 2016; Das et al., 2017; Xu et al., 2019], but enhanced by Transformer encoder. As Xu [2019] pointed out, the standard Key-Value Memory Networks contain the following components: key hashing, key addressing, value reading, query updating, and answer prediction, which also applies in our proposed method. And in the above components, the standard KV-MemNNs exploit the bag of words to encode queries, keys, values, and candidate answers. This straightforward approach is not appropriate to encode very long queries key-value pairs stored in the Legal Knowledge Base, as shown in a running example in Figure 4.

Encoders. Transformer has shown its ability to capture the long-range dependencies in natural language in GPT [Radford et al., 2018] and BERT [Devlin et al., 2019]. So rather

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

9

J

L

S

I

R

X

q

+

C

m

G

B

W

F

Y

>

A

n

c

V

N

8

E

3

U

r

/

0

K

o

M

T

Q

z

O

D

7

u

j

w

y

f

d

k

v

g

H

P

p

5

2

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

9

J

L

S

I

R

X

q

+

C

m

G

B

W

F

Y

>

A

n

c

V

N

8

E

3

U

r

/

0

K

o

M

T

Q

z

O

D

7

u

j

w

y

f

d

k

v

g

H

P

p

5

2

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

9

J

L

S

I

R

X

q

+

C

m

G

B

W

F

Y

>

A

n

c

V

N

8

E

3

U

r

/

0

K

o

M

T

Q

z

O

D

7

u

j

w

y

f

d

k

v

g

H

P

p

5

2

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

9

J

L

S

I

R

X

q

+

C

m

G

B

W

F

Y

>

A

n

c

V

N

8

E

3

U

r

/

0

K

o

M

T

Q

z

O

D

7

u

j

w

y

f

d

k

v

g

H

P

p

5

2

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

9

J

L

S

I

R

X

q

+

C

m

G

B

W

F

Y

>

A

n

c

V

N

8

E

3

U

r

/

0

K

o

M

T

Q

z

O

D

7

u

j

w

y

f

d

k

v

g

H

P

p

5

2

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

9

J

L

S

I

R

X

q

+

C

m

G

B

W

F

Y

>

A

n

c

V

N

8

E

3

U

r

/

0

K

o

M

T

Q

z

O

D

7

u

j

w

y

f

d

k

v

g

H

P

p

5

2

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

9

J

L

S

I

R

X

q

+

C

m

G

B

W

F

Y

>

A

n

c

V

N

8

E

3

U

r

/

0

K

o

M

T

Q

z

O

D

7

u

j

w

y

f

d

k

v

g

H

P

p

5

2

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

9

J

L

S

I

R

X

q

+

C

m

G

B

W

F

Y

>

A

n

c

V

N

8

E

3

U

r

/

0

K

o

M

T

Q

z

O

D

7

u

j

w

y

f

d

k

v

g

H

P

p

5

2

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

F

C

Y

0

+

B

o

y

u

k

d

N

2

A

5

P

M

n

K

>

c

V

7

T

w

X

q

R

g

J

E

U

Q

W

L

9

S

O

Z

I

D

j

3

p

v

f

8

r

G

/

z

m

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

F

C

Y

0

+

B

o

y

u

k

d

N

2

A

5

P

M

n

K

>

c

V

7

T

w

X

q

R

g

J

E

U

Q

W

L

9

S

O

Z

I

D

j

3

p

v

f

8

r

G

/

z

m

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

F

C

Y

0

+

B

o

y

u

k

d

N

2

A

5

P

M

n

K

>

c

V

7

T

w

X

q

R

g

J

E

U

Q

W

L

9

S

O

Z

I

D

j

3

p

v

f

8

r

G

/

z

m

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

F

C

Y

0

+

B

o

y

u

k

d

N

2

A

5

P

M

n

K

>

c

V

7

T

w

X

q

R

g

J

E

U

Q

W

L

9

S

O

Z

I

D

j

3

p

v

f

8

r

G

/

z

m

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

A

V

m

Z

z

C

Q

T

F

q

r

p

v

+

n

3

M

0

>

B

7

c

D

L

S

g

N

E

O

G

Y

I

y

K

o

8

R

J

P

w

k

9

X

d

2

u

U

f

j

5

/

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

A

V

m

Z

z

C

Q

T

F

q

r

p

v

+

n

3

M

0

>

B

7

c

D

L

S

g

N

E

O

G

Y

I

y

K

o

8

R

J

P

w

k

9

X

d

2

u

U

f

j

5

/

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

A

V

m

Z

z

C

Q

T

F

q

r

p

v

+

n

3

M

0

>

B

7

c

D

L

S

g

N

E

O

G

Y

I

y

K

o

8

R

J

P

w

k

9

X

d

2

u

U

f

j

5

/

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

A

V

m

Z

z

C

Q

T

F

q

r

p

v

+

n

3

M

0

>

B

7

c

D

L

S

g

N

E

O

G

Y

I

y

K

o

8

R

J

P

w

k

9

X

d

2

u

U

f

j

5

/

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

+

2

w

H

/

C

m

U

5

R

Z

n

X

g

y

J

V

F

8

E

M

>

A

B

c

N

S

3

r

q

9

L

0

K

o

T

Q

z

O

D

7

I

G

u

j

Y

W

f

P

k

d

p

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

+

2

w

H

/

C

m

U

5

R

Z

n

X

g

y

J

V

F

8

E

M

>

A

B

c

N

S

3

r

q

9

L

0

K

o

T

Q

z

O

D

7

I

G

u

j

Y

W

f

P

k

d

p

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

+

2

w

H

/

C

m

U

5

R

Z

n

X

g

y

J

V

F

8

E

M

>

A

B

c

N

S

3

r

q

9

L

0

K

o

T

Q

z

O

D

7

I

G

u

j

Y

W

f

P

k

d

p

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

+

2

w

H

/

C

m

U

5

R

Z

n

X

g

y

J

V

F

8

E

M

>

A

B

c

N

S

3

r

q

9

L

0

K

o

T

Q

z

O

D

7

I

G

u

j

Y

W

f

P

k

d

p

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

+

2

w

H

/

C

m

U

5

R

Z

n

X

g

y

J

V

F

8

E

M

>

A

B

c

N

S

3

r

q

9

L

0

K

o

T

Q

z

O

D

7

I

G

u

j

Y

W

f

P

k

d

p

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

+

2

w

H

/

C

m

U

5

R

Z

n

X

g

y

J

V

F

8

E

M

>

A

B

c

N

S

3

r

q

9

L

0

K

o

T

Q

z

O

D

7

I

G

u

j

Y

W

f

P

k

d

p

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

+

2

w

H

/

C

m

U

5

R

Z

n

X

g

y

J

V

F

8

E

M

>

A

B

c

N

S

3

r

q

9

L

0

K

o

T

Q

z

O

D

7

I

G

u

j

Y

W

f

P

k

d

p

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

+

2

w

H

/

C

m

U

5

R

Z

n

X

g

y

J

V

F

8

E

M

>

A

B

c

N

S

3

r

q

9

L

0

K

o

T

Q

z

O

D

7

I

G

u

j

Y

W

f

P

k

d

p

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

E

r

P

F

j

W

Z

f

K

3

v

I

V

9

J

2

o

>

A

B

n

c

N

S

8

q

/

L

0

m

M

C

T

Q

+

z

O

D

7

G

u

Y

w

y

X

k

d

H

g

5

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

N

O

M

U

D

g

Y

F

n

A

K

m

E

r

p

2

>

B

7

H

c

V

S

8

3

q

/

9

L

0

I

Q

Z

d

C

+

z

X

w

v

5

f

k

u

y

R

G

o

j

T

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

N

O

M

U

D

g

Y

F

n

A

K

m

E

r

p

2

>

B

7

H

c

V

S

8

3

q

/

9

L

0

I

Q

Z

d

C

+

z

X

w

v

5

f

k

u

y

R

G

o

j

T

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

N

O

M

U

D

g

Y

F

n

A

K

m

E

r

p

2

>

B

7

H

c

V

S

8

3

q

/

9

L

0

I

Q

Z

d

C

+

z

X

w

v

5

f

k

u

y

R

G

o

j

T

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

N

O

M

U

D

g

Y

F

n

A

K

m

E

r

p

2

>

B

7

H

c

V

S

8

3

q

/

9

L

0

I

Q

Z

d

C

+

z

X

w

v

5

f

k

u

y

R

G

o

j

T

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

N

O

M

U

D

g

Y

F

n

A

K

m

E

r

p

2

>

B

7

H

c

V

S

8

3

q

/

9

L

0

I

Q

Z

d

C

+

z

X

w

v

5

f

k

u

y

R

G

o

j

T

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

N

O

M

U

D

g

Y

F

n

A

K

m

E

r

p

2

>

B

7

H

c

V

S

8

3

q

/

9

L

0

I

Q

Z

d

C

+

z

X

w

v

5

f

k

u

y

R

G

o

j

T

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

N

O

M

U

D

g

Y

F

n

A

K

m

E

r

p

2

>

B

7

H

c

V

S

8

3

q

/

9

L

0

I

Q

Z

d

C

+

z

X

w

v

5

f

k

u

y

R

G

o

j

T

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

N

O

M

U

D

g

Y

F

n

A

K

m

E

r

p

2

>

B

7

H

c

V

S

8

3

q

/

9

L

0

I

Q

Z

d

C

+

z

X

w

v

5

f

k

u

y

R

G

o

j

T

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

+

8

S

K

r

d

J

5

n

o

7

G

c

D

z

w

>

A

B

V

g

N

E

O

M

H

Y

I

y

F

R

Z

p

P

T

Q

v

k

3

9

X

m

C

0

f

/

j

2

u

W

U

q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

+

8

S

K

r

d

J

5

n

o

7

G

c

D

z

w

>

A

B

V

g

N

E

O

M

H

Y

I

y

F

R

Z

p

P

T

Q

v

k

3

9

X

m

C

0

f

/

j

2

u

W

U

q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

+

8

S

K

r

d

J

5

n

o

7

G

c

D

z

w

>

A

B

V

g

N

E

O

M

H

Y

I

y

F

R

Z

p

P

T

Q

v

k

3

9

X

m

C

0

f

/

j

2

u

W

U

q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

+

8

S

K

r

d

J

5

n

o

7

G

c

D

z

w

>

A

B

V

g

N

E

O

M

H

Y

I

y

F

R

Z

p

P

T

Q

v

k

3

9

X

m

C

0

f

/

j

2

u

W

U

q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

+

8

S

K

r

d

J

5

n

o

7

G

c

D

z

w

>

A

B

V

g

N

E

O

M

H

Y

I

y

F

R

Z

p

P

T

Q

v

k

3

9

X

m

C

0

f

/

j

2

u

W

U

q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

+

8

S

K

r

d

J

5

n

o

7

G

c

D

z

w

>

A

B

V

g

N

E

O

M

H

Y

I

y

F

R

Z

p

P

T

Q

v

k

3

9

X

m

C

0

f

/

j

2

u

W

U

q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

+

8

S

K

r

d

J

5

n

o

7

G

c

D

z

w

>

A

B

V

g

N

E

O

M

H

Y

I

y

F

R

Z

p

P

T

Q

v

k

3

9

X

m

C

0

f

/

j

2

u

W

U

q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

+

8

S

K

r

d

J

5

n

o

7

G

c

D

z

w

>

A

B

V

g

N

E

O

M

H

Y

I

y

F

R

Z

p

P

T

Q

v

k

3

9

X

m

C

0

f

/

j

2

u

W

U

q

than encoding the input as a bag of words, we use Transformer. We append a special token, i.e., [CLS], at the tail of a sentence s, then input it into an encoder with n layer of Transformer encoder blocks. And then we use [CLS]'s corresponding output embedding enc[CLS] to represent the sentence s, and denote Transformer(s) = enc[CLS] in s.

Since the Transformer encoder are pre-trained on all queries, keys, values, and candidate answers with an autoregressive task of predicting next words, the encoding of [CLS] can not be directly used in the Question Answering task. So we apply a linear projection WLHS ∈ RH×H on Transformer(s), where H is the hidden size of Transformer encoder.

Equation (4) defines the Question Encoder in Figure 3, given the original query q as the input, to get the representation q(0) as the output. And all encoders in Figure 3 use a same Transformer.

Key hashing. This component searches on the whole Legal Knowledge Base and gets query-related key-value pairs by conventional IR methods, e.g., TF-IDF or SQL selection on specific fields. Because the IR method doesn't go deeper into the sentence-level semantics of the query, and key-value pairs, we need to store them in memory and do re-organization by the remaining components.

Key addressing and value reading. Key addressing reflects the relatedness between keys and query, as shown in Equation (5) and Equation (6). Similar to the scaled dotproduct attention in Transformer, we rescaled the relatedness score p(t) with a factor of 1/√H when doing normalization. The superscript t indicates the t-th hop (iteration) in multihop reasoning, and no more than the number T of total hops.

Query: . . . In a company, someone misappropriated 100,000 yuan of

public funds, according to Article [ARTICLE NUMBER] of the [LAW] ...

Answer: [ARTICLE NUMBER] = 272,

[LAW] = Criminal Law of the People's Republic of China

Related Keys & Values in Legal Knowledge Base

Key(1): Criminal Law of the People's Republic of China, Article 185

Value(1): Personnel of banks or other monetary institutions who take advantage of their office to misappropriate funds of their respective institutions or customers are to be sentenced and punished in accordance with the stipulations of Article 272 of this law. . .

Key (2): Value (1) Value (2): Key (1)

Key(3): Criminal Law of the People's Republic of China, Article 272

Value(3): When personnel of companies, enterprises, and other units, who take advantage of their offices to misappropriate their units' funds for their own use or for lending to others, and the amounts involved are relatively large and have not been returned for a period of over three months; . . .

Key (4): Value (3) Value (4): Key (3)

etc.

Candidate Answers:

Criminal Law of the People's Republic of China, Article 185; Criminal Law of the People's Republic of China, Article 272; etc.

Figure 4: A running example of filling slots with KB enhanced QA. The slots of [ARTICLE NUMBER], [LAW], and their context are treated as the query. The values of slots are the answer. In this example, the fact of misappropriating in a company leads to Article 272 as the answer instead of Article 185, because the latter is about the misappropriating in banks or other monetary institutions.

Value reading is given by Equation (7), summing the weighted encodings of values.

Essentially, key addressing and value reading implements an attention layer to select which values are used to answer the query q(t) .

Query updating. In the case of multi-hop reasoning, the query needs to be updated after each hop of reading on KB, because the newly read-in values should also be treated as a part of the query. In the running example in Figure 4, the query is to ask which law and article number applies to the fact of misappropriating in a company. After one hop, the fourth value in Figure 4 could answer the question. But in a more complex situation, such as the misappropriating in a bank, one hop is not enough because Article 185 also refers to Article 272.

We concatenate the query representation q(t) and the representation of related values o(t) , then map to the H dimension space by the matrix M (t) ∈ RH×2H to be a new query q(t+1) .

Answer prediction. Rather than picking up all keys in Legal Knowledge Base K as candidate answers like TextKBQA [Das et al., 2017], we only conduct the similarity computation between q(T) and the retrieved candidates by Equations (9) and (10).

Considering the article number and law in K plays three roles in the network, such as the keys, the values, and the answers, we use a different matrix WRHS to do the necessary decoupling in Equation (9). Finally, the loss of cross-entropy between the prediction score ˆ y(T) and the ground truth y is taken for training the Transformer-based key-value memory networks.

5 Empirical Study

In this section, we demonstrate the effectiveness of our proposed method CoLMQA in the following aspects. (1) We evaluate the quality of the generated texts with slots to show the language model is capable of predicting the number of slots and the composition of slots and words, which is critical to our divide-and-conquer strategy. (2) Comparing to other QA methods, we test the accuracy of filling slots by our proposed Transformer-based key-value memory networks.

5.1 Legal Document Dataset and Legal Knowledge Base

Legal Document Dataset. All the experiments are conducted on the judicial documents from China Judgements Online 2 , which are widely used in NLP works in legal domain [Luo et al., 2017; Ye et al., 2018]. We crawled 11,327,945 judicial documents from China Judgements Online in 2015. The legal documents cover diverse branches of law. For instance, we tokenized the document titles with a Chinese Tokenizer jieba 3 , and do the quick overview of the wordcount, then find out that the word "dispute" appearing in 4,774,968 (42.1%) document titles, the word "limited company" 30.5%, the word "criminal" 17.2%, and the word "divorce" 8.5%, etc.

We randomly select 20,000 documents from the dataset, splitting to the training data, validation data, and testing data, with the portion 80%, 10%, and 10%. By regular expressions, the fact descriptions, and court views are extracted. We mainly consider the generation of court views, which is the explanation for the charge. And court views contain the applicable law article to make the charge interpretable.

Legal Knowledge Base. We collect the required legislative documents according to the law articles mentioned in the legal document dataset, e.g., the Criminal Law of the People's Republic of China. On the basis of the collected legislative documents, the regular expression and a finite-state machine are used to extract the (key, value) pairs, where the key represents law article and value represents the detailed content. In the end, the Legal Knowledge Base is built upon the 149 laws, civil codes, and interpretations, which appeared in the aforementioned Legal Document Dataset.

2 http://wenshu.court.gov.cn/

3 https://github.com/fxsjy/jieba

Table 1: The F score of predicting law article slots (Type B slots) of different language models, on test dataset.

	without slots	with Type A slots	with Type B slots	with all slots
RNN	0.131	0.198	0.186	0.292
GPT	0.238	0.441	0.425	0.573
GPT (pre-trained)	0.421	0.642	0.573	0.682

Table 2: The perplexity on test dataset.

	without slots	with type A slots	with type B slots	with all slots
RNN	67.9	59.6	62.8	53.1
GPT	12.8	10.3	11.9	8.2
GPT (pre-trained)	11.3	8.8	10.5	7.8

5.2 LMwith Slots

Slots Based on the level of uncertainty, we modeled 7 slots: [NAME], [DATE], [MONEY], [ADDRESS], [NUMBER], [LAW], [ARTICLE NUMBER]. We use the regular expression to replace the values in court views part in legal documents with the above slot placeholders. The first five slots reduce the uncertainty of language modeling. The last two slots correspond to two parts of a law article, which are the law's name and the number pointing to the law article. They have specific values in our built Legal Knowledge Base. These values can be obtained by reasoning based on the text content. Therefore, they are within the consideration of slot filling in our question-answering model. And we denote the first five slots as the Type A slots and the last two slots as the Type B slots. The Type A slots used here is to demonstrate that language models can benefit from reducing the uncertainties when generating text, especially when they don't have to predict the exact number of [MONEY] or the value of [DATE].

Language models with(out) slots. Language Models are categorized into different groups depending on whether trained with slots or without slots as shown in Table 1. We use two versions of GPT 4 , pre-trained, and without pre-trained. The former is pre-trained on a news dataset 5 which contains 2.5 million news articles for one epoch on a 4× NVIDIA Tesla V100 machine, and with the stride being 768. The sequence length of the Transformer block used in both GPT versions is 1024. The text generation sub-task is given a prompt (the facts part in a judicial document), then to generate a sentence by using the Beam search.

4 https://github.com/huggingface/transformers

5 It's news2016zh dataset in https://github.com/brightmart/nlp chinese corpus.

Result. In the first sub-task, after removing some highly uncertain content and replacing it with slots, the language model can generate more deterministic text and make the sentence more fluent as shown in Table 1 and Table 2.

Given a prompt, we use a language model to generate the court view that contains slots and then count the generated Type B slots in it. The F score in Table 1 is computed between two Type B slot sequences by matching one by one. As a toy example, a ground truth in a court view containing slots like "in accordance with Article b and Article b of a", where a=[LAW], b=[ARTICLE NUMBER], and the slot sequences in a generated text is like "in accordance with Article b of a, Article b of a, and Article b of a". Then the precision of predicting Type B slots ([LAW] and [ARTICLE NUMBER]) is 1/6 because the first one slots are matched one by one, while the recall of predicting Type B slots is 2/3 when considering the total length of subsequences "b*a" in the ground truth is matched by the generated sequence. The F score in this toy example is 0.27. In this way, the F score is used to measure how good the text sequence with slots is matched to the ground truth. When computing the F score of the slots prediction by using the model without slots, we do a manual conversion to replace the values with slots in the result.

In Table 1, we compare different language models and different settings and find out that the pre-trained GPT can effectively generate a sentence with correct slots. In Table 2, the perplexity on the test dataset demonstrates that the generated sentence of the court view is fluent when considering slots.

5.3 Filling Slots by Transformer-based Key-Value Memory Networks

Construction of query-answer pairs. To derive the queryanswer training pairs from the Legal Document Dataset in Subsection 5.1, we consider different situations as follows.

The first one is that a court view part in the legal judicial document only contains one applicable law article in the form of "According to Article 263 of the Criminal Law of ...". In this case, the query in a query-answer pair is the text span from the beginning to the position of current law article, while the value is the ground truth, e.g., Article 263 of the Criminal Law. The second one is that a court view contains more than one applicable law article with the same law, such as "According to Article 263 and Article 269 of the Criminal Law". In this one, we make multi tiny-modified copies, Query1: "According to [ARTICLE NUMBER] of the [LAW]", Query2: "According to Article 263 and [ARTICLE NUMBER] of the Criminal Law". The third one is the mixing case, that different laws and different law articles are cited in a court view, which can be processed like the second one. On average, there are 4.1 query-answer pairs produced per court view. And the average length of a query is 2783. For the query which is longer than the sequence length of Transformer Encoder in TKVMemNN, we keep the tail part of the query sequence, which is reasonable as the fact description appears more often in the tail part.

Training of our method. We use the first four layers of Transformers in our pre-trained GPT as the encoder for queries, keys, values, and answers in the experiment, then carry out the two-phases fine-tuning. The first-phase finetuning is conducted on the task of predicting the next words in the sequences of queries, keys, and values. The second-phase fine-tuning is training the proposed model TKVMemNN on our constructed query-answer pairs and the Legal Knowledge Base. To conduct the multi-hop reasoning, we set the number of hops T as 3.

Baselines. We use Whoosh (a Python IR open-source toolkit), Memory Networks, and their variants with or without the document title as baselines. As the title are important for providing additional information, such as the type of documents, they are useful for filling slots.

Because the computation of the dot product between the query and the encodings of all the candidates in the Legal Knowledge Base is very expensive, we limit the number of candidates in IR scope to improve the efficiency.

Table 3: The accuracy of filling slots.

MemNN	Whoosh	Whoosh +title	TKVMemNN	TKVMemNN +title
0.23	0.13	0.17	0.39	0.41

Result. In the second sub-task, the multi-hop reasoning on the knowledge base makes the generated text more logical as illustrated in Table 3.

6 Conclusion and FutureWorks

In order to solve the problem of conflicting logics appeared in the text generated by a language model, we propose a novel method CoLMQA which benefits from Language Modeling and Question Answering simultaneously. In the Language Modeling component, the sentences with slots are generated and provided a well-written skeleton of sentences. And in the Question Answering component, the slots are filled with accurate values with the help of Knowledge Base. In our experiment, we verified that our method can fill slots with logical coherent values. In the future, we'll expand CoLMQA to other scenarios of text generation which also require keeping logic, but using different kinds of slots.

Acknowledgments

We thank Xi Chen, Ni Li, Yan Cui, Mingkuo Ji, Jun Chen, Yuren Wu, and Hanzhang Yang for helpful discussions.