Hybrid Thematic Role Processor: Symbolic Linguistic Relations Revised by Connectionist Learning

1 Introduction

In sentences such as

(1) The man broke the window with the stone,

one can intuitively find an AGENT (the man), a PATIENT (the window), and an INSTRUMENT (the stone). Linguistic theory [Haegeman, 1991] refers to the roles words usu­ ally have in relation to a verb as thematic roles, so that one can say that break has a thematic structure with the following roles [AGENT, PATIENT, INSTRUMENT), in this sentence. But linguistic theory also assumes that this structure can change, depending on the sentence. So, for the sentence

(2) The stone broke the vase, Edson Fran~ozo LAFAPE- IEL- Unicamp Caixa Postal 6045 13081-970 - Campinas - SP- Brasil edson@iel.unicamp.br there is a different thematic structure, since stone is CAUSE (the one that causes the action) and vase is PATIENT. The difference between (1) and (2) is that al­ though the same verb is employed (break), no AGENT or INSTRUMENT is expressed in (2); thus, the thematic structure for (2) - [CAUSE, PATIENT] - is different from the thematic structure for (1).

The theoretical approach to thematic roles in linguis­ tics is symbolic. As in predicate logic, the linguistic ex­ pressions are decomposed into a central predicate (often the verb) and a number of arguments that complete its meaning [Raposo, 1992]. The predicate assigns thematic roles to the arguments, and each sentence has a thematic grid, i. e., a structure with all thematic roles assigned to the sentence arguments by the predicate.

A Natural Language Processing system, called HTRP (which stands for Hybrid Thematic Role Processor), is proposed to identify the thematic grid of a semantically sound input sentence. Two versions are deployed : the first, without initial knowledge, and the second, with initial knowledge. The first version specifies an ordinary connectionist architecture with initial random connection weights, henceforth called RIW (random initial weight version). In the second version, henceforth called BIW (biased initial weight version), a set of biased initial network connection weights is introduced to represent symbolic rules for ten thematic roles. In both versions, after supervised training, a set of final symbolic rules is extracted, which is consistently correlated to linguistic (symbolic) knowledge. In the case of BIW, this amounts to a revision of the initial rules. In RIW, symbolic rules seem to be induced from the connectionist architecture.

2 Thematic Roles

Taking sentences ( 1) and (2) again, it seems that the distinction between AGENT and CAUSE has something to do with the nouns that are assigned such roles. Thus, since only an animate noun is supposed to be an AGENT, some kind of semantic analysis is necessary in order to distinguish between different thematic assignments . In other words, thematic roles must be elements with semantic content [Dowty, 1989]. One could then imagine that the words, which can fill each of the slots for a given thematic grid, share a common semantic core. Assuming this is regular, one could try to capture such regularity (a) by describing each word in terms of its semantic features, and (b) by generalizing over all such descriptions for each thematic slot.

Semantic feature generalization is the hallmark of McClelland and Kawamoto's [1986] pioneering proposal, and of much subsequent work. In a system called CPPro [Rosa, 1997], a connectioni·st architecture based on an adaptation of McClelland and Kawamoto's [1986] model is proposed. The words are represented by arrays of se­ mantic microfeatures, formed by subsets accounting for aspects of word meaning, like human and non-human, where only one value in each subset is active. For the verb, these arrays are arranged on the basis of thematic relationships between the verb and the other words of a sentence, thus mapping thematic roles onto semantic features. The aim of CPPro is to explore the idea of mi­ crofeature representation in order to build an architecture able to analyze and to learn the correct thematic relation­ ship attributions of the words in a sentence. Its output reflects judgements of semantic acceptability of a sen­ tence.

In HTRP, the output is constituted by the thematic grid of a sentence, composed of up to ten thematic roles : AGENT, EXPERIENCER, CAUSE, PATIENT, THEME, SOURCE, GOAL, BENEFICIARY, VALUE, and INSTRUMENT. For HTRP, some intuitive thematic role definitions are adopted, as follows . AGENT is the argument having the control of the action expressed by the predicate. EXPERIENCER is a participant who does not have the con­ trol of an action expressing a psychological state. CAUSE is the argument that initiates the action expressed by the predicate without controlling it. PATIENT is the partici­ pant affected directly by the action of the predicate, usu­ ally changing state. THEME is the participant affected indirectly by the action of the predicate, without chang­ ing state. The other role labels are self-explanatory.

In HTRP, only sentences with up to three arguments are taken care of. Thus, the argument structure [Haege­ man, 1991] of the sentences is as follows :

where arg 1, arg2 and arg3 are the arguments of the verb, to which the predicate (the verb) assigns thematic roles. A limited set of verbs is chosen for the present imple­ mentation of HTRP: break, buy, deliver, fear, frighten, give, hit, and love.

2.1 Verb Representation

The representation of the verb in HTRP is strongly based on Franchi and Can~ado [1998]. They use a non­ lexicalist representation; that is, the thematic role assignment compositionally depends on the whole sen­ tence. For instance, taking the verb break, (5) and (6) are the thematic grids for (3) and (4) respectively:

1. Mary broke the vase with a hammer.
2. The stone broke the vase.
3. [AGENT, PATIENT, INSTRUMENT)
4. [CAUSE, PATIENT) .

To explain the difference, one can resort again to the notion that thematic roles are elements with semantic content. In this case, it seems that sometimes (e.g. in sentence (3)) control of action is required by the verb break in relation to arg 1, while no such control is re­ quired in sentence (4). Thus, one could say that control of action is a feature to be associated with the verb.

The same is true for the verb frighten, regarding a different feature: direct process triggering.

1. Mary frightened Paul with a scream.
2. The tests frightened Paul.

In (7) control of action is part of the game, while in (8) direct process triggering assumes a central role .

Thus, a small set of features can be associated with the verb, in the same manner that nouns are associated with a set of (different) features [Waltz and Pollack, 1985; McClelland and Kawamoto, 1986; Rosa, 1997].

The compositional features associated with the verb change according to the sentence in which the verb is used. So, it is inadequate to say that a specific verb has a single thematic grid, because this will depend on the whole sentence in which the verb occurs. In sum, a non­ lexicalist approach is preferable.

3 The Connectionist Architecture

HTRP system uses a connectionist architecture repre­ senting eleven independent artifi.cial neural networks, one for each thematic role and one for the error output [Law­ rence et al., 1999]. The elementary processors are classi­ cal perceptron-like units, and each net has 40 input units, 2 hidden units, and one output unit. The input units are responsible for the representation of two words of a sen­ tence, the verb and one of the nouns. Since each HTRP sentence has, at most, three nouns beyond the verb, each sentence works with at most three neural networks, in order to activate a grid of up to three thematic roles. The first hidden unit (V) represents the conjunction of all the verb microfeatures, and the second (N), the conjunction of all the noun microfeatures. The output unit represents the conjunction of these two microfeature sets (see figure 1 ). The error output, which has also two hidden units and one output unit, differs at the input layer, which in this case has 80 units, because it is unknown which nouns, in conjunction with the verb, activate the error output.

3.1 The Error Output

Lawrence et al. [ 1999] propose a recurrent neural net­ work to classify English sentences as grammatical or ungrammatical, exhibiting the same discriminatory power supplied by linguistic theory. The network is not divided into innate and learned knowledge. Instead, positive and negative examples are used to discriminate between grammatically acceptable and unacceptable sentences.

In HTRP, an error output network is implemented, in order to account for this. For a semantically unacceptable input like:

1. The stone bought the man

the system activates the error output. So, before generat­ ing the thematic grid for a sentence, HTRP tests the se­ mantic acceptability of such sentence, so that the system only reveals the thematic grid for semantically well formed sentences.

4 The Hybrid Approach

Since its inception, Artificial Intelligence (AI) is torn between two opposing fields : the symbolic paradigm, based on logic, and the connectionist paradigm, based on the propagation of the activity of elementary processors.

Artificial neural networks do not have the expressive power of general logical representations, since they are not adequate for manipulation of high level symbols [Fodor and Pylyshyn, 1988] . They are usually preferred in a number of situations (such as pattern recognition) because they are able to generalize over the inputs, they are fault tolerant, and they exhibit the ability to learn from experience.

But neural networks have a disadvantage: often be­ cause of lack of transparency it is hard to understand how they build their inner representations. For instance, it is not easy to ascertain the meaning of the connections and their weights or the configuration of the hidden layers as regards a certain input-output pair.

But, the so-called knowledge-based neural networks, which bring the opposing AI paradigms into closer con­ tact, allow for symbolic knowledge to be introduced in as well as to be extracted from neural networks - that is called hybrid approach.

The extraction of symbolic knowledge from trained neural networks permits the exchange of information between connectionist and symbolic knowledge repre­ sentations and has been of great interest to understand what the neural network actually does [Shavlik, 1994]. Additionally, a significant decrease in learning time can be obtained by training networks with initial knowledge [Omlin and Giles, 1996] . Also, the symbolic knowledge can be input into neural networks and then refined after training.

In the hybrid approach adopted here, the symbolic knowledge is represented through connection weights between neural network processing units. For instance, a fuzzy logical rule, with weighted antecedents A and B, and consequent C,

can be represented by a connectionist schema, as shown in figure 2. The rule is fuzzy-like, because wAc and w8c (connection weights) are not binary values but real num­ bers. Also, it simulates an and unit, such that only the presence of both inputs A and B causes unit C to be acti­ vated.

The symbolic knowledge generated by the net can be extracted, in both versions of HTRP, in a way compara­ ble to initial symbolic knowledge implementation in BIW, using the above structure.

4.1 Microfeatural Representations

Word representation in HTRP is adapted from the se­ mantic microfeature representations used by Waltz and Pollack [1985] and McClelland and Kawamoto [1986], for the noun. For the verb, the representation is mainly based on Franchi and Can~ado [1998]. Nouns and verbs are accounted for by twenty binary semantic microfeature units each. The following general schema represents the nouns:

• human - non-human
• soft- hard
• small - medium - large
• 1-D/compact- 2-D- 3-D
• pointed- rounded
• fragile/breakable- unbreakable
• value - furniture - food - toy - tool/utensil - animate

For each of these subsets, only one feature is active, and all the others are inactive. For instance, man is hu­ man, soft, large, 3-D, rounded, unbreakable, and ani­ mate; stone is non-human, hard, small, 3-D, pointed, unbreakable, and tool/utensil.

The system also includes ambiguous nouns, so that some of its microfeatures are undetermined. In such cases, the system will arrive at the missing values for the intended reading, because it is fault tolerant.

The following schema represents the verbs:

• arg 1 has control of action- no control
• direct process triggering - indirect triggering
• direction to source (argl)- direction to goal (arg2)
• impacting process for arg2 -no impacting process
• change of state of arg2 -no change of state
• psychological state- no psychological state
• argl has objective- no objective
• effective action- no effective action
• high intensity of the process - low intensity
• arg I has interest on process- no interest on process

Again, for each of these subsets, one feature is active, and the other is inactive. For instance, in the sense of sentence (l) above, for break the following features are active: control of action, direct process triggering, di­ rection to goal, impacting process, change of state, no psychological state, objective, effective action, high in­ tensity, and interest on process. In the sense of sentence (2), the following features are active: no control of ac­ tion, indirect process triggering, direction to goal, im­ pacting process, change of state, no psychological state, no objective, effective action, high intensity, and no in­ terest on process. As one can see, two different readings for the same verb break.

But when the user enters the verb break into HTRP, the system does not know which break is intended. And, the network input is the "average" of the two readings of break. Again, some of the microfeatures will be undeter­ mined. And again, the system will arrive at the missing values for the intended reading of break.

4.2 Initial Symbolic Rules

The HTRP "thematic rules" inspired by Haegeman [1991] and McRae et al. [1997] for 13 types of verbs (8 different verbs and 5 alternative readings) were also im­ plemented. · The rules are if-then rules (logical implica­ tions), and they are implemented as an and gate, i. e., if an input is absent, the unit should not be activated. Un­ like classical logic, each element in the antecedent part of the rules is weighted in a fuzzy way, by the connection weight of the respective element in the network. Then, for a unit to be active, all its inputs together should be such that their sum is enough to activate the unit (see figure 2).

For each thematic role there are two 'hidden' rules whose antecedents map the units belonging to the input layer and whose consequents map hidden units - one for the verb, and the other for the noun (see figure 1 ). For instance, for the thematic role AGENT in BIW, there is no initial rule for the noun (N), because any noun can in principle be an AGENT. The system, after learning, will decide which nouns could be AGENTs. But for the verb (V), the rule is:

If for verb (0.2 control of action) + (0.2 direct process triggering) + (0.2 impacting process) + (0.2 objective) + (0.2 interest on process)

Then V

If (0.5 V) + (0.5 N) then thematic role = AGENT.

4.3 The Learning Step

The training sentences are generated by a sentence gen­ erator, alternating verbs and nouns. Both semantically sound and ill-formed sentences are generated. For BIW, learning begins after the introduction of initial symbolic rules as connection weights of the network. The algo­ rithm used is the supervised backpropagation [Rumelhart et al., 1986] . After 3,000 training cycles, the system is able to judge, with a high degree of certainty, if a sen­ tence is meaningful or not, and, if it is, which its thematic grid is.

One interesting consequence of learning is that the system is able to categorize on the basis of the comple­ mentarity of the verb microfeatures for most subsets. Consider the system without initial knowledge (RIW); in this case, the initial connection weights for each subset of microfeatures are random. Since. during training the sen­ tences exhibited mutually exclusive values within each subset of microfeatures, the final connection weights are found to be complementary in the sense that their respec­ tive values are of opposite signals . That is, the network incorporates the complementarity of microfeatures in virtue of its architecture and experience.

4.4 Final Rules

Rule extraction consists in reversing the process of initial rule insertion, in BIW. That is, the net weights are as­ sessed and a weighted antecedent is obtained, corre­ sponding to the connection weight. This rule is fuzzy because it allows for weighted antecedents in the pro­ duction rule. The symbolic knowledge thus extracted from the present connectionist architecture corresponds to the network learning and generalization capacities. As a consequence, the network is able to "revise" the initial symbolic rules. The fuzzy rule extraction from the net­ work, after training, for both versions of HTRP is based on Fu [1993], Setiono and Liu [1996], and Towell and Shavlik [1993].

For RIW, the final rules for the thematic role AGENT are the following:

'Hidden' rules:

If for verb ( -0.6 control of action) + ( -1.0 direct process triggering) + ( -0.1 direction to goal) + ( -0.9 impacting process) + (-1.1 change of state) + (-0.1 no psychological state) + (-2.2 objective) + (-0.6 effective) + (0.2 high intensity)+ (-0.8 interest on process)

Then V

If for noun (1.7 human) + (0.2 soft) + (3.1 me­ dium + 1.8 large) + (0.2 3-D) + (0.2 rounded) + (1.4 unbreakable) + (3.7 animate)

Then N

'Output' rule:

If ( -7.3 V) + (6.9 N) then thematic role= AGENT.

Notice that almost all antecedents of the 'hidden' rule are negative for the verb. But the antecedent of the 'out­ put' rule (-7.3 V) is also negative for the verb, which means that the negative signals cancel each other out.

Notice also that, for the verb, many microfeatures were highly biased by learning: control of action, direct process triggering, impacting process, change of state, objective, effective, and interest on process. In relation to the noun rule, the AGENT learned by the net is mainly medium and animate, and less prominently, human, large , and unbreakable.

For BIW, that is, with the introduct}on of initial sym­ bolic rules, for the thematic role AGENT there is the fol­ lowing final rule for the verb:

'Hidden' rule:

If for verb (0.9 control of action) + (1.2 direct process triggering) + (0.8 direction to goal) + (0.5 im­ pacting process) + (0.4 change of state) + (0.1 no psy­ chological state) + (1.2 objective) + (-0.1 effective) + (0.2 high intensity)+ (1.2 interest on process)

Then V

As one can see, considering the initial rule antece­ dents, all features were highly strengthened by learning, with the exception of impacting process, which rose only from 0.2 to 0.5. That is, the system can be said to rein­ force the initial features .

There is a final rule for the noun too:

'Hidden' rule: If for noun (-1.6 human) + (-0.3 soft) + (-2.3 medium + -0.8 large) + (-0.7 3-D) + (-0.6 rounded) +

(-0.6 unbreakable)+ (-2.6 animate)

Then N

'Output' rule:

If (7 .1 V) + ( -7 .I N) then thematic role = AGENT.

Since the 'output' rule shows a negative antecedent for the noun ( -7 .1 N), all the negative weights of the 'hidden' rule antecedents become positive. So, the AGENT learned by the net is mainly human, medium and animate, and, with less prominence, soft, large, 3-D, rounded, and unbreakable.

Notice that there are small differences between the final hidden rules for nouns in RIW and BIW, although one might expect them to be the same because both for BIW and RIW no initial rules for nouns are provided. Such difference stems from (i) the connectionist archi­ tecture employed, which takes into account both verb and noun inputs to activate the thematic role output (see fig­ ure 1); and (ii) from the backpropagation algorithm, which causes verb weights to influence noun weights during the error backpropagation step.

To illustrate and compare the differences between RIW and BIW, a summary of the weights for the verb, concerning the thematic role AGENT, is presented in table 3. Recall that these values are used to weigh the micro­ features in the antecedents of the symbolic rules.

Note: Mf = semantic microfeature; ca = control of action; dt = direct triggering; dg = direction to goal; im = impacting process; cs =change of state; np =no psychological state; ob = objective; ef = effective; hi = high intensity; ip = interest on process; I = initial weights; FR =final weights for RIW; FB =final weights for BIW.

Table 3. A comparison between initial and final weights.

Notice that, when initial knowledge is input to the system (BIW), there is a tendency of strengthening the initial weights. When no initial knowledge is provided (RIW), the final weights are quite close to those in BIW. This can only be taken as evidence that the final weights reflect the available symbolic knowledge (about a the­ matic role) from the examples and from the architecture, since in this case the initial weights are arbitrary.

5 Conclusion

In the realms of connectionist Natural Language Proc­ essing, several systems use the notion of thematic role modeling (e.g., McClelland and Kawamoto [1986], McClelland et al. [ 1989], St. John and McClelland [1990], Jain [1991], and Miikkulainen [1996]). Also, at least one recent paper [Chan and Franklin, 1998] imple­ menting a hybrid system makes use of the notion of case roles, which is close to the concept of thematic relations. The present system departs from all these in that it relies on the role of semantic entailments in thematic relations, i.e., in the way it makes use of theoretical knowledge from linguistics.

HTRP implements a symbolic-connectionist hybrid approach to thematic role processing. In this approach, the advantages of symbolic systems (ease of knowledge representation, understanding through logical inference, etc.) are combined with the advantages of connectionism (learning, generalization, fault tolerance, etc.) to yield a more discriminating thematic role processing, that is sensitive to the subtleties involved in such linguistic phenomenon.

The representation of semantic features adopted in this system would also easily allow for new words to be entered in order to increase its lexicon, once their se­ mantic microfeature arrays are supplied. In HTRP a sin­ gle network accounts for each verb-noun pair; thus gen­ eralizing over both nouns and verbs. In fact, this is cru­ cial in dealing with thematic roles, for they are but the generalization of semantic relationships between verbs and nouns. Another interesting result that should be em­ phasized regards RIW. Even in a system without initial knowledge, the final rules extracted from the network fully correspond to the symbolic theory that explains them. That is, it seems that the HTRP architecture to­ gether with training is enough for the system to arrive at the correct semantic grid of a sentence.

Acknowledgements

We thank Marcio L. de Andrade Netto, Ester M. Scarpa, and Plfnio A. Barbosa, of Unicamp, for their extensive comments and inspiration on drafts of this paper. We also thank IJCAI'99 reviewers, for their very useful sugges­ tions. Of course, remaining mistakes are our own.