Meaning and the Mental Lexicon

1 Introduction

The mental lexicon refers to the representations that al­ low word recognition on the basis of auditory and vi­ sual stimuli. The lexicon is understood as two linked levels of representation: The first level consists of form­ based representations that reflect a word's phonological or graphemic properties. The second level contains se­ mantic representations that reflect its meaning relations with other words [Marslen-Wilson, 1989].

Priming studies are an important source of evidence for the semantic organization of the lexicon. When sub­ jects are presented briefly with a letter string, followed by another, and asked to decide whether the latter is a real word, the response time when both strings are related is reliably faster than when they are unrelated. Priming effects can be found using stimuli that are graphemically, morphologically, or semantically related ([Taft, 1991] for a review) .

*The author is supported by a Medical Research Council studentship.

Substantial progress has been made modelling the form-based lexical representations in the light of graphemic or phonological similarity [Plaut et al., 1994], but there is currently no principled measure of seman­ tic similarity. Word meaning is much more difficult to quantify.

The network described here is an attempt to address this problem. It is inspired by two relatively indepen­ dent approaches to semantic representation from cog­ nitive psychology and computational linguistics. After considering each approach I describe the network's im­ plementation and present results. The next section de­ scribes the structure and development of semantic repre­ sentations, and how the model relates to previous work. Finally I consider the model's psychological relevance with reference to developing categorizations and semantic priming.

2 Lexical-semantic Networks

A highly influential theory of lexical-semantic represen­ tation from cognitive psychology is based on the seman­ tic network . A semantic network consists of a set of nodes and connections of varying strengths , or lengths, between them [Collins and Loftus, 1975]. Each concept is assigned a node, and connection strengths reflect the amount of conceptual relevance each node has to its part­ ner . The stronger, or shorter, connections represent a high level of similarity. Weaker, or longer, connections hold between less related nodes. In a lexical-semantic network (LSN) , each node represents a word and the distance between nodes reflects the amount of semantic similarity between each word. The Logogen model [Mor­ ton, 1979], Interactive-activation model and spreading activation accounts, are all types of LSN [Neely, 1991].

LSN accounts explain semantic priming effects in the following way: Each node has an activation level. When a stimulus is presented it activates all nodes in the net­ work to some degree. If one node is activated strongly enough its activation will pass a threshold and fire. The stimulus will be recognized as that word. Each time a word is presented, activation spreads from the most acti­ vated node to nearby nodes, decaying over time. For ex­ ample, if 'doctor' is presented shortly before 'nurse', the node associated with 'nurse' will reach threshold faster and fire sooner. Its resting activation level is raised by activation spreading from 'doctor' during the inter­ stimulus interval.

3 Data-intensive Semantics

Recent work in computational linguistics suggests that large amounts of semantic information can be extracted automatically from large text corpora on the basis of lexical co-occurrence information [Lund et al., 1995; Schiitze, 1993]. This approach is particularly well suited to neural network implementation [Finch, 1993] because co-occurrence statistics track conditional probabilities, and neural networks have straightforward interpreta­ tions as statistical models [Bishop, 1995].

The data-intensive approach to semantics is consistent with, and inspired by theories of meaning that empha­ size the importance of use [Wittgenstein, 1958] (see also [Church and Mercer, 1993]). Lexical co-occurrence in­ formation reflects a word's distributional profile, which is a reflection of its use.

The success of the data-intensive semantics research shows that, with a large enough sample, there is suf­ ficient information in a strictly linguistic environment to recover much semantic structure. It seems plausible, therefore, to investigate the possibility that the brain makes use of such information. The recent discovery that semantic and associative priming effects in the lexical de­ cision task are significantly correlated with co-occurrence statistics [Lund et al., 1995; Spence and Owens, 1990] support this possibility. Lund et al. constructed a high­ dimensional space on the basis of lexical co-occurrence counts. Words that were close together in the space gave larger priming effects than those further away.

4 Modelling the Lexicon

LSN theories provide an intuitive way to understand word meaning and its relation to priming. However, there is no theory of how the nodes of a network are formed, or how the distance (or strength) relations be­ tween them become organized.

The data-intensive approach to semantics is an ef­ fective predictor of semantic priming, and reflects an influential approach to understanding word meaning. However, the approach requires an extremely high­ dimensional co-occurrence space for lexical-semantic rep­ resentation. It is not obvious how such a space could be represented in the brain.

The model presented below is a first attempt at ex­ plaining how the semantic level of the lexicon could be organized, consistent with the LSN and data-intensive semantics approaches, in a way that is computationally tractable and biologically reasonable.

4.1 Overview of the Model

The model consists of an input layer that picks out words from a text stream, a dynamic proto-lexicon which records co-occurrences between the present target word and words either side of it , and a self-organizing map. The proto-lexicon is initially empty and the self­ organizing map weights are set to random values.

4.2 Implementation

Proto-lexicon

The proto-lexicon represents each word in terms of the number of times it has been seen to co-occur directly be­ fore and after each other word in the vocabulary. Specifi­ cally, in ann-word vocabulary each word Wi (1 ~ i ~ n) is associated with the vector xi = [6, . . . , 6n] normal­ ized to unit length, where ~j (1 ~ j ~ n) denotes the frequency with which Wj has preceded Wi before t, and ~k (n+ 1 ~ k ~ 2n) denotes the frequency with which Wk has succeeded Wi. Thus at each time step, xi represents the model's best guess for the conditional probabilities

and

for all words j, k and timet. Each successive xi is an improved estimation of the true distributional profile of each word.

In large-scale applications it is usual to distinguish a fixed subset of high-frequency words to serve as con­ text (see [Church and Mercer, 1993] for a review). Co-occurrence vectors calculated using high-frequency words are less sparse and provide better samples. This technique complicates the relation between the co­ occurrence vectors and quantities (1) and (2), though the results are robust to approximation. The model is presented without approximation.

Self-organizing Map

The self-organizing map is presented with the current proto-lexical representation for each word as it is en­ countered in the text stream. The winning unit is the unit i* with weight vector w;. such that

for all i :j:; i*. Output unit weights are updated after each word presentation using a variation of Kohonen's self-organizing map algorithm [Kohonen, 1982]:

where TJ is the learning rate and 4>(i, i*) a neighbourhood function. i represents the unit to be updated, and i* the winning unit . 4>( i, i*) takes the value 1 when i = i* and decreases as a function of the distance betwe~n i and i* according to:

During the course of training rr and TJ are slowly re­ duced 1 .

Input

Input consists of words taken sequentially from a 20,000 word corpus generated by a stochastic context-free gram­ mar described in [Elman, 1990] . A sample section of input is shown below.

man like boy lion eat mouse

Each time a word is recognized in the text stream, its representation in the proto-lexicon is updated and pre­ sented to the self-organizing net as an training item.

4.3 Results

After moving through the corpus once, clusters of seman­ tically related words emerge. Each word is presented to the network the winning output unit is recorded and la­ belled. Figure 1 shows the winning unit for each word after 20,000 word presentations.

Consistent with the LSN approach, the network clus­ ters each word with other words that are used in similar contexts. Similar words tend to be nearer to one another than to dissimilar words. Verbs have been represented together on the right side: Psychological verbs 'see' and 'smell' are represented together, as are destructive verbs 'smash' and 'break' within the main verb group. On the left side, categorial similarity among the nouns is equally well preserved - human and animal nouns group separately, adjacent to one another.

However, figure 1 does not reflect the full extent of the net's categorization. 'Man' is equidistant from 'boy' and from 'book', but is related much more closely to one than the other. This fact is represented by the network, not in the pattern of winning units, but by the pattern of activation across all output units. Figures 2,3 and 4 show activation plots for 'man', and for 'boy' and 'book' with the unit specialized for 'man' marked. 'Boy' is asso­ ciated much more strongly with 'man', than with 'book' because 'boy' is the highest unit on a plateau contain­ ing all the human nouns, whereas 'book' is in a separate region shared by inanimate nouns.

Figures 5 and 6 also show how the network creates a hierarchical semantic representation: 'See' and 'smell'

1 u( t) = u; ( ;t-) ..-n!ar where u; represents the initial setting and O"j the final, value. ry(t) = ry;(*)..-n!ar where '7i and etai represent the initial and final learning rates, respectively. For these simulations u; = 20, u 1 = 0.5, ry; = 1 and '11 = 0.01.

share an activated region but this region is a section of the larger region that covers all verbs.

5 Structure and Development of Semantic Representation

Structure

Although figure 1 resembles a classical semantic net (without connections), unlike many other LSN models, it uses a distributed coding scheme; the activation lev­ els of all units are used to express semantic relation­ ships among words. Distributed coding is both more reliable and more biologically realistic than classical lo­ calist coding. It is more reliable because representations need not be compromised by the loss of single units . In topographic maps, if a unit specialized to some in­ put pattern is lost, one of its neighbours will become the winner for that pattern on subsequent tests, but the shape of the output map will remain largely intact due to graded transitions in feature-specificity across neigh­ bouring units. Graded transitions between cell response profiles due to topographic organization have been re­ ported in many brain regions [Knudsen et al., 1988].

In the model graded transitions between winning units also allow uncommitted units to capture new words with distributions similar to more than one word in the initial vocabulary.

The data-intensive approach to semantics explains se­ mantic similarity in terms of points in a high-dimensional space. By the end of training, the proto-lexical repre­ sentations define such a space. Here 29 words create a 58-dimensional space. In order to form an output repre­ sentation of the type required by the LSN approach, and to explain how such a space could be represented in the more limited dimensions of the brain, this space must be reduced to a more manageable size. Two properties of the self-organizing map algorithm make it especially well-suited to this task:

1. The self-organizing map algorithm creates a non­ linear projection from a collection of data points in a high dimensional space defined by each input vector to a one or two-dimensional grid of output units. The projection attempts to preserve the topology of the input space in the lower dimensional output space. Thus the algorithm performs precisely the data-reduction necessary.
2. The algorithm has a straightforward physiological interpretation: it models the development of fea­ ture selectivity due to lateral inhibition among cor­ tical nerve cells [Sirosh and Miikkulainen, 1993; Kohonen, 1993]

It is possible to pinpoint cell groups relevant to nam­ ing and semantic memory tasks using electrode map­ ping techniques during neurosurgery [Ojemann, 1983;

1991]. In a review of language localization studies, [Bradshaw and Mattingley, 1995] conclude that lexi­ cal representation probably depends on many areas dis­ tributed across the cerebral cortex. This is consistent with the structure of the model.

Development

During the course of training, output units become in­ creasingly sensitive to a particular input pattern. The weight update equation ensures that the weight vector of the winning unit and its neighbours are moved toward the input pattern. However the input patterns are also moving. Representations in the proto-lexicon alter each time a new word is read. This means that different words get different amounts of training devoted to them. Dur­ ing development, the region over which updates occur is shrinking. This entails that input patterns that are far from one another in the input space are separated first on the output map, and patterns that are very close to each other are separated later [Kohonen, 1982]. If the algo­ rithm is interrupted before completion, the final output representation provides a usable partial categorization of the input data (See also section 7).

6 Related Work

The self-organizing map algorithm has been used suc­ cessfully to model semantic clustering [Miikkulainen, 1993], but Miikkulainen's networks require a static set of input patterns to be constructed by a separate extended­ backpropagation system with theta-role assignments, be­ fore clustering can begin. In contrast, this system oper­ ates with raw text, without staged processing, and with­ out supervised components. [Ritter and Kohonen, 1989] have also used self-organizing maps, but used heavily pre-processed input data, and a much smaller corpus.

Elman's Simple Recurrent Networks [Elman, 1990] also make use of of co-occurrence statistics, represented in the hidden node activations. However, in the Simple Recurrent Network, hidden unit response profiles must be manually extracted and submitted to cluster analysis before semantic similarity information becomes directly available. In the model presented here co-occurrence in­ formation is explicit.

7 Psychological Relevance

The model presented here illuminates two psychologi­ cal phenomena: learning to distinguish between seman­ tically similar words, and semantic priming effects.

Discrimination

The shape of the output map reflects the network's in­ creasing ability to distinguish words that are semanti­ cally distinct. The semantic difference between 'mouse' and 'sandwich' is greater than between 'mouse' and 'cat'.

The developmental schedule of the network reflects this, by learning to distinguish 'mouse' from 'sandwich' be­ fore distinguishing it from 'cat'. While 'cat' remains undistinguished the two words are essentially the same to the system. The discriminative capacities of the sys­ tem also depend upon exposure; high frequency words are distinguished sooner in general. These observations are consistent with research into child language acqui­ sition, suggesting that broad semantic distinctions be­ tween frequent items are discovered first [Harris, 1992]. [Finch and Chater, 1994] have shown how partial cate­ gorizations may be used to bootstrap more complex rep­ resentational structure during language development .

Semantic Priming

The network described above is a type of LSN. Conse­ quently it is possible to formulate an account of semantic priming within this framework . With the current archi­ tecture each word that is encountered activates the out­ put layer independently, via its proto-lexical representa­ tion. To allow an explanation of priming it is necessary to relax this restriction and assume that word recogni­ tion advances all unit activations to the levels particular to the recognized word in small amounts over a brief time period, and that unit activations then decay over time until the activation surface is flat.

The explanation of semantic priming is then straight­ forward: If a prime word is presented before activation levels have fully decayed, then residual activation will still be present in some units. For example, let 'man' be the target word, and let the related word 'boy' and the unrelated 'book' be primes. 'Man' (fig. 2) will take longer to be recognized as a word when preceded by 'book' (fig. 4) than when it is preceded by 'boy' (fig. 3) because the activation surface for 'man' is almost iden­ tical to that of 'boy', but quite different from the activa­ tion surface for 'book'; fewer increments are necessary to convert the activation surface for 'boy' into the surface for 'man' than to convert the surface for 'book' into the surface for 'man'.

This account of priming is similar to other LSN mod­ els. Priming effects depend on the distance between the prime and target words because the the map is organized such that activation tends to drop off with distance from the winning node.

8 Conclusion

The network presented here models the formation and arrangement of the semantic level of the mental lexi­ con. It is consistent with the lexical-semantic network approaches to lexical arrangement and semantic prim­ ing, and with the data-intensive approach to semantics. The network represents semantically similar words to­ gether using lexical co-occurrence information that is calculated as the network moves through a text corpus. The network uses a topographic mapping technique that is widespread in the brain, and provides a biologically reasonable account of mental lexicon.

Acknowledgments

I would like to thank my supervisors Richard Shillcock and Mark Ellison, and Joanna Bryson for encouragement and much useful discussion.