Computer Simulation of Articulatory Activity in Speech Production

Introduction

In modeling the process of human speech production we recognize five dis­ tinct stages. First, the message is organized on the linguistic level and structured grammat ical ly. Second, the message i s expressed phonologically in terms of a sequence of discrete units, l abelled phonemes. Stress and intonation marks can be considered special phonemes and assigned special codes. Third, the string of phonemes is converted to a set of continuous signals controlling the musculature of the various articulators . This results in a highly complex inte­ grated movement sequence in which gener­ ally all the art iculators, the lips, the tongue, the mandible, etc., participate. Finally, the physical interaction of the vibrating vocal cords and the moving arti­ culatory structure produces a continuous acous tic signal perceived as speech.

We are particularly interested in modeling the fourth or the articul atory movement stage of the speech event. We cons ider the speech event as composed of a sequence of phonetic segments specified in terms of ·one or more vocal-tract shapes and the corre sponding excitation para­ meters. The acoustic signal is generated by simul ating the excitation mechanism a s well as the articulatory trajectory. In contrast, most previous speech production systems consider generation of the speech s igna l from the acoustic point of view and are controlled by specifying the variations of vocal-tract resonant frequenc i es (formant frequencies) with time.l,2 Modeling the process a t the art i culatory level can be expected to be simpler be cause the a rticulators respond to muscular forces with predictable changes in their position and rates of movement. On the other hand, simple rules for resonant frequency changes are not neces­ sarily realizable by the anatomy of the vocal tract. In fact, s imple articula­ tory changes can have complex acoustic effects. This report is primarily con­ cerned with generating the acoustic signal corresponding to given changes in the shape of the vocal tract. Accomplish­ ment of that task is a prerequisite to studying the movements actually exe cuted by the articulatory system when producing a particular utterance.

Recent studies show that no phoneme, not even the vowels, exhibits a unique set of resonant frequencies in all its occurrences in connected speech.3 There­ fore, it is unreasonable to expect that a unique vocal-tract shape can be associ­ a ted with all productions of that phonema Certain attributes or distinct i ve fea­ tures of the excitation mechanism and of the articulatory state serve, however, to differentiate among the phonemes. Such features may include presence or absence of labial closure, of periodic or turbu­ lent excitation, coupling of the nasal pas sage to the main vocal tract, etc. While incorporation of these features in the articulatory de scription is necessary to obtain an intelligible result, it is not sufficient. The human speech percep­ tion apparatus is easily confused by f alse cues that arise from unnatural art i culations . Therefore, those para­ meters unspecified by the distinctive fe atures of the phoneme must be specified with reference to the moving a rticulatory structure s o that the result will cor­ respond to the natural execution of the art ic ul at ion. Contextual effects such as antic ipation or ine rt i a control these optional fe atures, and one i s interested in specifying in detail the nature of

t hAt control mechan i sm.

If the effects of context are expressed in rules for a rticulatory move­ ment, only the essentia l feature s of the phonemes need be spec ified expl i c itly, and the optional parameters will be deter­ mined automatically. The simulation sys­ tem is intended to aid the postulation of such rules by serving as a convenient tool for the evaluation of their a rticulatory and acoust ic effects . This report de­ scribes a facility for specifying a dis­ crete sequence of excitation and art icula­ tory states of the vocal tract, and the system simulates the articulatory movement between these states, thus producing a continuous acoustic signal . The segment corresponding to a particular phoneme is represented as a sequence of one or more such states, where the numerical descrip­ tion of the states may be functions of the adjacent phonemes. For example, sustained tongue tip closure in the absence of voic­ ing is a feature of the consonant /t/, but the shape of the tongue hump for that pro­ duction is controlled by the adjacent vowels. Aspiration, another context­ dependent feature of the / t/, is achieved by optionally introducing states specify­ ing the onset and termination of turbulent excitation. For the moment, all para­ meters of the excitation and vocal-tract shape must be specified explicitly, or they are assigned fixed default values .

The timing of articulatory movement can be controlled in open-loop or closed­ loop modes. The closed-loop mode would require continuous position or velocity feedback; however, the rapidity of the articulatory movement does not allow for this. Hence we assume a ballistic-type movement based on the present and the target articulatory positions. The time interval for the movement is explicitly specified, and it is assumed that the forces generated by the musculature are adjusted accordingly . The time durations are in practice functions of the adjacent articulatory states and future models can be expected to compute them from the arti­ culatory specifications . Where timing differences arise from causes other than the art iculatory context, such as the duration difference in English vowels before voiced and unvoiced consonants, these cannot be handled at the art i cula­ tory level and would have to be introduced explicitly even where articulatory move­ ment proceeds completely by rule.

The speech production appara tus is modeled as consisting of a periodic source of variable frequency and ampl itude, the vocal cords, and/or a noise source, excit­ ing a nonuniform acoustic transmission line cont inuously changing in shape. The shape, of course, controls the character­ i st ic impedance at any point, and thereby the transfer function of the line. Arti­ culatory states are represented in terms of sampled values of the vocal- tract cross-sectional area at points along the tract .

4 Henke has postulated a set of rules governing tongue, lip and mand ible move­ ments based on analyzing frames from high­ speed X-ray motion pictures of the moving vocal tract. However, he did not evaluate the resulting speech signal aurally. Our data are derived from acoust ic measurement of the vocal-tract cross- sectional area5 and admit the postulation of similar rule& This report is primarily concerned with the implementation of a f acility for imme­ diate aural evaluation of utterances gen­ erated with the aid of postulated rules . Comparison of the computed trajectories of the vocal-tract resonant frequencies as functions of time with those measured from natural articulatory events yields a quantitative difference measure that can be used to complement the subjective evaluation of the audio signal . In addi­ tion, the specified movements are dis­ played for evaluation as sagittal projec­ tions convenient for comparison with X-ray pictures.

Static electric analogs of the vocal tract6,7,8 have been used for an extensive period as tools in the study of how defor­ mations in the vocal tract control the characteristics of the s2eech signal. Dynamic electric analogsY,lO have been used for the synthesis of a restricted number of syllables, but they suffered from an inability to control the transmis­ sion-line elements over wide ranges at high speeds. Kadokawa and Nakatall have simul ated the dynamic behavior of the vocal tract by computing a time - dependent transfer function and thus determining the var i ations with time of the vocal-tract resonant frequencies. Coker12 uses atime­ varying parametric representation of the vocal tract to compute the changing reso­ nant frequencies and thus control a for­ mant synthesizer . The primary obstacle to the development of a general synthesis f ac ility has been the unavailability of a suitably flexible control system that allows quick and convenient spec ification or change of the parameters controlling the tract configuration. The computer­ s imulated system not only eliminates limi­ tations to rapid configuration changes over a wide range of cross-sectional area values, but also permits experiments designed to search for convenient control strategies.

The presence of the human listener in the sound generation - evaluation - modification - sound generation feedback loop is necessary because at present we are unable to ass i gn effective objective criteri a to the significant attributes of the acoustic result. Naturalness i s a subjective criterion, and therefore the presence of the listener i s essential. The experimenter also acts as a visual observer , evaluating shape and resonant­ frequency variations with time. We en­ deavor to fac ilitate the experimenter's task so that in the light of his evalua­ tion he can change the data var iables qu i ckly and conveniently. This is accom­ pl ished by allowing the experimenter to control the simulation using measures familiar to him, for example, lip opening or excitation frequency. He may build up the data for an utterance item by item, evaluating the result each time. Alter­ nately, he may modify data previously entered and hear the modified audio signal in less than a minute.

The Simulation Method

The simulation principles are based on those used by Kelly and Lochbauml3 with cons iderable extensions and modifications. The vocal tract is represented ·as a non­ uniform transmission line supporting longitudinal propagation only. Its char­ acteristic impedance at all points is inversely proportional to the cross­ sect ional area in a plane perpendicular to the flow line. The tract is assumed hard­ walled; the compliance of the soft tissues of the tract wall is ignored. The sampl­ ing frequency for the simulation, 24 kHz, is determined by the spacing of area samples, 1.5 em, that is necessary to simulate the continuous tract up to a signal frequency of 6kHz. Since the length of each section is considerably smaller than the wavelength even at this frequency, the effect of the distributed reflect ion of a section traversed by the wavefront in one sampling period can be represented by a lumped reflection coeffi­ c ient centered on the section .

Figure 1 shows a block diagram of the vocal-tract model. Pressure samples pro­ pagate back and forth along the transmis­ sion line and suffer multiple interna l reflections . Excitation may be periodic, simul ating vocal-cord vibration, or tur­ bulent corresponding to frication. For periodic excitation the samples of the excitation function are added to the samples of the reflected signal arriving at the glottis. Turbulent excitation may take place at the vocal cords or at any interior section. It is implemented by means of a white - noise generator and associated sour ce impedance placed in series with the line. For computational s implicity, the tract is assumed lossless within its interior. Simple digital filters act at the tract boundaries, the vocal cords and the lips, to approx imate the frequency-dependent losses so that actually measured resonant-frequency band­ widths a re properly matched. The glottal reflect ion and the l abial r adiation func­ tions act essentially as high-passfilters; the labial reflection appears as a low­ pass filter. The nasal passage is repre­ sented by an add itional transmission line of fixed but nonuniform shape coupled to the main tract with the aid of a variable coupling parameter . It serves in the pro­ duction of nasalized vowels and consonants. The system output is obtained by summing the outputs of digital filters approximat­ ing the free-space radiation character­ istics at the lips and nostrils . Radia­ tion through the lips is, of course, con­ trolled by the variable lip area . If the opening is reduced, more ~f the energy arriving at the lips is refle cted within the tract and less is radiated to the outside. Radiation through the tract wall is not explicitly included.

Articulatory movement is simulated by defining a sequence of articulatory states or targets and interpolating the position­ dependent reflection coefficient values and oral-nasal coupling between them. Thus targets are not necessarily phonemic in nature but may be specified as frequently as desired in order to approx i­ mate natural articulatory movement as closely as possible. When the articulator positions are derived from a model of articulatory activity, the vocal - tract shapes computed therefrom are expected to act as the target states of the simulation program. Stress and intonation informa­ tion are assumed embedded in the excita­ tion frequency and amplitude parameters and will not be discussed here in detail .

Linear interpolat ion of the reflection-coefficient values implies that near sharp constrictions the area var ies l inearly with time, but where the area changes slightly or the reflections are small, the areas vary exponentially with time . The resonant -frequency curves resulting from this procedure resemble those observed in spectrograms more closely than the straight-line segments used in most formant synthesizers .

Implementation

The system is operational on a DDP-224 computer. All of the routines are written in FORTRAN, except for the most frequently executed loop of the wave pro­ pagation computations which was rewritten in assembly language. Among the note­ worthy hardware features utilized is the use of input / output channels to feed the loudspeakers through the digital - to-analog converter from one memory module, while simultaneously reading the next record from tape or disk into a second module .

Cathode-ray tube display proceeds simul ­ taneously with processor operations on an interrupt basis.

The execution time of the simulation is approx imately 40 times real time. The sample propagation computat ions along the transmission line, although basically very simple, are the most time consuming. A special slave processor has been designed to carry out these operations under con­ trol of the general purpose computer. If implemented, it would cut the simulation time to four times real time. Immediate application to real-time speech synthesis is therefore limited and the system is considered primarily a research tool.

The excitation period serves as a time clock for the synthesis routine, and within each period parameters or parameter increments rema in constant. Therefore an excitation period is specified even when the tract is excited by noise or not excited at al l as i n pauses within the utterance . This alllows a relatively rapid, free-running simulation for hundreds of sampling periods before a check is made for new parameter values .

Once the control data have been specified the simulation runs without interaction, and wr ites the speech output on disk or tape. Thus the time duration of speech material generated in any run is limited only by the control table storage limitat ions and the patience of the experimenter. Because the pressure dis­ tribution along the vocal tract at any time within the utterance cannot be simply recreated, a change in any one target parameter requires the regeneration of the complete utterance.

Control and Use of the Simulation Facility

The control parameters for a part i­ cular synthesis task are specif ied in terms of two parameter lists . The first is a list of art iculatory states defined in terms of sets of 13 area values . The second list is a time sequence of targets, nasal coupling values, target interval durations, excitation fre quency values, excitation amplitude value s , whether peri­ odic or random excitation and if the l atter , the place of excitation . Targets are spec ified by using character-string l abe ls as pointers to entries in the table of articul atory states . Excitation para­ meters are conveniently separated from specification of the art i culatory conf i gu­ rations and each is easily varied inde­ pendently of the other. The same articu­ latory configuration may be repeatedly called by its label in the second list without having to spec ify it again in numerical terms . The program suppl i es default values for most parameters, gener­ ally unchanged from the previous specifi­ cation so that only changing parameters need be explicitly specified . Targets may be appended, inserted or deleted from the list conveniently and quickly. A typical sentence, the example discussed later, which consists of 19 phonemes required a sequence of 27 articulatory targets for acceptable synthesis .

The basic mode of data entry is through the on-line typewriter . Once a parameter list satisfactory to the experi­ menter has been entered, the program gen­ erates the output signal and records it digitally on tape or disk. Because the simulation runs slower than real time, the output signal is converted to analog form and played back only after its generation has been completed .

The above method of operation, though perfectly feasible, is not terr i bly con­ venient from the point of view of an experimenter not used to numerical speci ­ fication of articulatory data. To visu­ alize the art iculatory configurations and movements between them, a time-motion dis­ play of the art iculatory trajectory is projected at a rate proportional to the real - time movement rates specified . Any particular conf igurat ion may be selected for static display, graphically modified with the aid of a visible pointer, and returned to its slot in the target history of the utterance .

Another display that has found wide application to evaluate the course of articulatory events is one where the sequence of vocal-tract resonant frequencies are displayed on a time base . To ease the computational l oad, the reso­ nances are computed from the stored shapesl3 rather than the generated s ignal . This display resembles a conventional spectrogram in appearance. Based on his wide experience with spectrograms, an experienced speech scientist can rapidly pin-point s ituations where the acoust i c effects of the specified articul atory tra­ jectories deviate appreciably from normal patterns. The v isual i nformation he obtains thus complements h is aural evalu­ ation of the generated utterance .

Let us now consider in detail simula­ tion of the production of the sentence, "This is a story about a man." This sentence was generated through trial and error procedures by modifying shapes derived from X- ray pictures and referring to the natural utterance for timing valuea It demonstrates the quality of output available through careful work with the system. No explicit articulatory rule s were used i n this example . Instead the experimenter's accumulated knowledge of articulatory dynamics and the acoust ics of speech production were utilized. The aut omat ion of this process s o that one will not have to resort to trial and error for every different sentence generated is the next task at hand.

In Figure 2 the spectrograms of the synthetic and natural utterances are com­ pa red. We transcribe that sentence in terms of phonemic symbols

/~1 s l z a 'stor1 ·'bavt ~ 'm~n/ . The apostrophes indicate stress marks which are considered in specifying the excita­ t ion frequency and target interval dura­ tion data. The excitation frequency func­ tion for the sentence is obtained by modi­ fying the function determined from the intonation pattern. The excitation frequency is increased prior to and de­ creased relatively more rapidly after stress marks. For the voiced fricatives /~/ (th) and /z/ we supply, in add ition to periodic excitation, turbulent excitation a t points 3 em and 1.5 em resp~ctively behind the front end of the tract. In each case the vocal-tract shape is rela­ tively constricted at that point. The unvoiced fricative /s/ is excited by noise only at the same point and using the same vocal-tract shape as /z/ . The vowels /~, /~/, / o/ and / ae / are periodically excited and have their articul atory configurations shaped by modifying the configuration appropr i ate for the isolated vowel in accordance with the articulatory influ­ ences of the neighboring consonants. In turn, the vowels are used to control the variable articulatory characteristics of the consonants . For example, the articu­ l at ion of the initial / s / is the same as that of the / Z/ except where the tongue tip i s r a ised to produce the constriction. The glide / r / is vowel-like in excitation and retroflexion of the tongue is not applied . The diphthong /aU/ is gene r ated by linear reflection-coefficient inter­ polation between bounding target values that are somewhat modified versions of the isolated independent vowels /a/ and /U/. The unvoiced stop /t/ has a transition part to an articulation exhibiting almost complete constriction at a point 3 em from the front. This constricted articulatory state is sustained for a 21 msec interval before release . On release a short burst of noise is introduced to simulate affr i­ cation or turbulence; .thereafter the amplitude of the periodic excitation is linearly increased from a zero value. The voiced stop /b/ has the articul at ion changing from that of the previous vowel /~/ to that of the initial part of the following diphthong /aV/ while the lips are kept almost completely constricted . A very small but finite lip opening is help­ ful in s imulating the acoustic effects of energy radiated through the walls of the mouth. The nasal consonants / m/ and /n/ a re produced with the oral tract com- . pletely constricted at the lips and 3 em behind them, respectively, but with a high value of nasal coupling . The coupl i ng is reduced but not eliminated for the i nter­ vening /ae I vowel which is thus made nasa­ lized .

One interesting result observed through trial and error experiments is that naturalness is enhanced by avoiding sustained articulations - those where two adjacement art iculatory states are spec i­ fied as identical. Except for the vowels in the stressed syllable s and the frica­ tives, the art iculations are continuously changing . This further i nvalidates the idea that speech can be produced by con­ catenat ing different stationary signals .

Conclusions

The simul at ion program produces intelligible speech if care if taken to modify articulatory shapes , time dura­ tions, and excitatlon frequency values to fit the contextual environment on a level at least as large as the sentence. Data appropr i ate for the production of an iso­ l ated word but not for the sentence i n which it is embedded may make not only that word, but the complete sentence unintelligible. The quality of the mate­ rial generated so f ar can be used as a goal for the quality to be achieved by implementing rules governing the movements of the individual art i cul ators.

The program uses an excitation func­ tion that is stored and independent of the vocal-tract shape. The articul ation, how­ ever, is known to affect both the wave­ shape and the excitation frequency.l5 The fixed wave - shape independent of frequency i s but a rough approximation to the real situat ion. Detailed s imulat ion of vocal­ cord movement would, however, increase the execution time of the program by at least a factor of two . Because these factors a re not central to the aspects of speech generation under consideration at the moment, they have not been included in the present program.

The vocal tract is assumed lossless in its interior, therefore, the resulting resonance bandwidths are fixed functions of frequency and independent of the arti­ culat ion. The losses in general do depend on the surface area of the tract wall and are thus affected by the shape of the tract in the plane perpendicular to the flow directions . The contribution of this factor to lack of naturalness has not been precisely evaluated.

Articulations of / 1/ and / r / , in particular, often exhibit sidebranches formed by the tongue and the hard palate that cannot be modeled precisely when one treats the vocal tract as one continuous tube . Our approximations are therefore expected to lead to differences in the acoustic result and must be recognized as limitations of the model.

A program that requires expl i cit specification of all articulatory and exc i tation parameters suited to the parti­ cular context cannot be said to exhibit extensive artificial intelligence. Instead, the simulation program must be viewed as a framework within which state­ ments regarding articulatory movements can be expressed and evaluated. The framework reported is based on the consideration that utterances are organized on the arti­ culatory level, and that organization can be studied most effectively at the same level.

Our results up to this time concern organization at the lowest level. Here articulatory parameters are treated as independent, and interactions between them, except as reflected on the acoustics level, are ignored. For example, the nasal coupling parameter is used to con­ trol nasalization of vowels adjacent to a nasal consonant at the same time as the tongue shape is changing as required for the production of those vowels. Thus the adjacent vowels are perceived as nasa­ lized. The production of the same effect on the acoustic level is much more com­ plex. This example substantiates the basic motivation for speech generation through articulatory control, that this direction of attack can unravel the com­ plex organization of the individual pho­ nemes into the integrated utterance.

References

1. Holmes, J. N., Mattingly, I. G. and Shearme, J. N., "Speech Synthesis by Rule 4 " Language and Speech, 7, 127 (196 ). -
2. 2 . Rabiner, L. R., "Speech Synthesis b~ Rule: An Acoustic Domain Approach, ' Bell System Tech. Journal 47, 17 (1968). -
3. 3 . Lindblom, B., "Spectrographic Study of Vowel Reduction," J . Acoust. Soc. Am., 35, 1773, (1963).
4. Henke, W., "Dynamic Articulatory Model for Speech Production Using Computer Simulation," Ph.D. Thesis, Mass. Inst. of Tech., Cambridge, Mass., 1966.
5. Schroeder, M. R., "Determina tion of the Geometry of the Human Vocal Tracts by Acoustic Me asurements," J . Acoust. Soc. Am ., 41, 1002 (1967 ) .
6. Dunn, H. K., "The Calculation of Vowel Resonances and an Electrical Vocal Tract," J. Acoust . Soc . Am., 22, 740 (1950).
7. Stevens . . K. N., Kasowski, S. and Fant, C. G. M., "An Electric Analog of the Vocal Tract," J. Acoust. Soc. Am., 25, 734 (1953 ) .
8. Fant, C. G. M., Speech Communication Research, Ing. Veitenshaps Akad., (Stockholm), 24, 331 (1953)
9. Rosen, G., "Dynamic Analog Speech Synthesizer," J. Acoust. Soc. Am., 30, 20 (1953).
10. Hecker, M. H. L., "Studies of Nasal Consonants With an Articulatory Speech Synthesizer," J. Acoust. Soc. Am., 34, 179 (1962).
11. Kadokawa, Y. and Nakata, K., "Analy­ sis of Speech by Vocal-Tract Configu­ ration," J. Radio Res. Laboratories, Japan, ll, 99 (1964) .
12. Coker, C. H., "Speech Synthesis with a Parametric Articulatory Model," paper A-4, Speech Symposium, Kyoto, Japan, 1968.
13. 13 . Kelly, J. L., Jr. and Lochbaum, C. C., "Speech Synthesis," Proceedings of the Fourth International Congress on Acoustics, Copenhagen, 1962.
14. Mermelstein, P., "Determination of the Vocal-Tract Shape from Measured Formant Frequencies," J. Acoust . Soc. Am., 41 , 1283 (1967).
15. Flanagan, J. L. and Landgraf, L. L., "Self-Oscillating Source for Vocal­ Tract Synthesizers," Trans. IEEE AU-16, 57-64 (1968).

N 6

:I:

~ 5

z

>- 4

LLI

0 z 3

:::>

0 2

a:

LLI

La.

MACHINE GENERATED SENTENCE

NATURAL SENTENCE

• 2 . Spectrograms of machine generated and natural vers ions of "This is a story about a man. "

2

2

by

B. L. Hills, R. L. Beurle & M. V. Daniels,

Department of Electrical and Electronic Engineering, University of Nottingham, England.

Sunnnary

Measurements of threshold contrast required for various resolution tasks at various back­ ground light levels are reported. The predic­ tions of a simple statistical model based on photon-noise limited detection ~re compared with the empirical observations. Correspondence is encouraging provided care is taken to account for the variation of all important parameters such as spatial and temporal integration, pupil area, etc.

Introduction

Since the early work of K5nig in 1879, it has been known that visual acuity, i.e. the ability to discriminate fine detail in an object, pro­ gressively deteriorates as the level of illumi­ nation falls. Shlaer 1 has shown that this fall in visual acuity is dependent on the type of acuity pattern used. Thus at background luminances above 100 trolands, the gap in a 'Landolt C' pattern is easier to see than bars of a similar width in a parallel bar resolution pattern. At lower luminances, the reverse is true. The ultimate limit of acuity under optimum viewing conditions depends critically on the nature of the task. For vernier acuity, a displacement of 2 seconds of arc between the top and bottom halves of a thin vertical black line has been detected. For the Landolt 'C' and the parallel bar pattern, gaps and bar widths of the order of 25' are the best that have been resolved.

It is generally agreed that the fall in visual acuity at low levels of illumination is connected with the changeover from using the small cone summation areas of the fovea at high adapta­ tion levels, to using the larger rod summation areas of the periphery at low levels. Also, these summation areas are known to increase in size with decrease in the background level, to which the eye is adapted. This will also reduce acuity.

This account is concerned with the recog­ nition of simple patterns of varying contrast at relatively low light levels and is an extension of work on detection previously described by the authors 2 · The empirical observations of detec­ tion threshold described in this reference have been compared with the predictions of a simple statistical model for a photon-noise limited detector. This work, which is to be published elsewhere, showed an encouraging correspondence, and the present paper describes the application of the same technique to resolution thresholds.

The Ideal Detector

The original idea of comparing the actual performance of the human visual system with an 'Ideal Detector' is due to Rose3, although similar theories were developed independently by de Vries and also Pirenne. An ideal detector (or ideal picture pickup device) is one whose performance is limited only by the statistical fluctuations in the number of incident photons picked up by the device. Presented with the task of detecting a pattern distinguished by a small luminance, ~I, superimposed on a uniform background of luminance I, this detector can do no better than count the photons arriving within the area where the pattern is anticipated, and compare this with the mean density of arrival of photons in the background I. Background photons are considered indistinguishable from target photons, and all these incident photons are given equal weight in the output of the device (i.e., it is linear), so that every effectively absorbed quantum is taken into account in the final decision.

In adapting this concept, the ideal detector has been assumed to be subject to certain limi­ tation analogous to those of the eye. For example, it is taken to have the ability to integrate temporally for a period which is assumed to vary as empirical observation suggests it does for the eye. Then again it is assumed that there is spatial integration over an area the extent of which depends on the background light level. Some assumptions must be made about the method of determining I. For example, it may be assumed that the background can be sampled on a sufficient number of occasions or over a suffic­ ient area by the ideal detector for the back­ ground luminance I to be known precisely. Alternatively, it may be assumed that the measure­ ment of I is subject to error in the same way as

the measurement of I+~I.

Now, it is generally accepted that the num­ bers of absorbed quanta fluctuate according to a Poisson distribution, the deviations from an average absorption of n quanta thus having a root mean square value of n°·5· Thus if n quanta are absorbed from the background, it is these fluctuations that interfere with the detection of a small change ~I in illumination. If this small change in illumination, or 'signal', yields ~n absorbed quanta then, for a given reliability, the threshold of detection is given by

~n = k n°' 5

··············· (1)

where k is a constant called the threshold signalto-noise ratio, its exact value depending on the required degree of certainty in detection. Equation (1) is the basic equation of the fluctuation theory of Rose. It is, however, an approximation, for it ignores the fluctuations in the signal. Thus, a more rigorous form of the equation is For all except the lowest background levels, this equation approximates to (1), since l'ln >> n, but at very low background levels when this condition does not apply, equation (2) must be used.

In the experiments on detection, it was shown desirable to assume that in what has been called a primary recipient unit 2 the sensitivity varies radially over the effective summation area over which spatial integration takes place, a fact which has been recognised in the literature 4 · 5· The assumed variation is shown in Figure 1, 2 ·5 this curve representing a radial variation of the form

where rm is a function of the background luminance, as shown in Figure 2. The adoption of this expression has been justified merely for con­ venience in calculation. It gives a good fit to the range of empirical results which it represents, but there is no reason to think that it has any biological significance. The summation time T and pupil area, ~P' were assumed to vary with background luminance in accordance with the curve shown in Figure 3 and the data given in Table I.

With this knowledge of the spatial and temporal summation characteristics of the retina, it now becomes possible to calculate the number of photons l'ln in a single sample taken by the eye from a pattern illuminated by I'll. The mean number of quanta n from the background I may similarly be calculated. Substitution of these in equation (1) now gives an equation for I'll in terms of background luminance I. As a rough approximation this is of the form

but the precise relationship is appreciably affected by the variation with light level of rm, T and ~p· Since the technique of calculation is summarised in more detail in a companion papers , it is not proposed to elaborate further on the details here.

It was found that for large objects at higher light levels the primary recipient detector was inadequate to explain the results, and it was essential also to assume the existence of elongated linear "edge" detectors2 · The evidence suggested that these had a transverse variation of sensitivity similar to the variation across the diameter of the units already referred to. This could be the result of combining the output of a linear array of primary summation units. There was also evidence for a drop off in sensitivity towards the ends of these long summation areas and a law similar to equation (3) was assumed, replacing rm with a length constant Lm which was also found to be a similar function of background level. Values assumed for all these parameters are given in Table I which also gives ap the area of the natural pupil assumed in these calculations.

Since the detailed results of these experi­ ments are to be published elsewhere, it will suffice here merely to say that for the patterns tested, discs, annuli and parallel bars, the correspondence between empirical thresholds· and predicted thresholds was encouraging, discrepan­ cies being typically less than 0·2 in logarithmic units to the base 10.

Resolution Measurements

The experiments reported here were designed to compare the threshold contrast for resolution at various light levels of the following pattern shapes:

Parallel bars The "Foucault Fan" pattern The double disc pattern The 'Landolt C' A square equal to the gap in the ~andolt C' A vernier acuity pattern

The shapes of these patterns are shown in Figure ~ Three different sizes of each pattern were used, and the object was to see whether the variations between resolution thresholds for different patterns could to any extent be accounted for by the application of the photon noise theory out­ lined above.

In these experiments the subjects were given adequate time to adapt to an evenly illuminated background, and the resolution pattern was super­ imposed on this, the colour temperatures being 3600°K and 3400°K respectively. The resolution thresholds were determined by the subject, who had control of the illumination of the pattern. He was asked to straddle the point at which the resolution feature of the pattern was just visible. It is thus possible to make a distinction between the "detection" threshold at which it is possible to see the pattern as a whole, and the recognition threshold at which is is possible to see the reso­ lution feature (e.g. the bars in the parallel bar resolution pattern. With the exception of the square all the patterns have such a feature, e.g. the gap in the "Landolt C", the existence of two separate discs and the discontinuity in the verti­ cal line. The square was included for comparison with the Landolt C where the task might be des­ cribed as "seeing" the "missing square" where the gap is present.

The Results

The averaged increment thresholds for the three subjects are given in Table II. For simplicity, the thresholds for the patterns will be compared two at a time.

One simplifying feature of the results is that the corresponding bar and fan patterns (as defined in Figure 4) have virtually identical increment thresholds. This is true for all three sizes and all background luminances. This is because the pointers indicating where 'z' is measured are centrally placed on the fan pattern. It has been found that if the pointers are used too near the top, the increment threshold for the fan is higher than for the bars. Thus, with the proviso that the pointers are used in the cen~ral region, the comparisons of performance that w1ll be made with the parallel bar pattern would be the same if the comparisons had been made with the fan pattern.

In Figure 5, the first comparison is made between the increment thresholds for resolution of the Landolt 'C' and the parallel bar patterns. It is first noted that the increment thresholds for the bar patterns are considerably lower than the equivalent Landolt 'C' as defined in terms of z in Figure 4. There are major diff~rences between the thresholds for these two patterns. For the large and medium sizes of both patterns, the increment threshold progressively decreases as the background level is lowered, but for the small patterns, the increment threshold is constant for background luminance~ of 10- 4 mL and below.

It will be seen that vertical displacement of the curves for the bar pattern by 0·7 log units gives reasonably close fits to the curves for the Landolt C. This is true to some extent for all the patterns, excluding the squares, and might suggest that once some allowance has been made for the overall inefficiency factor of each pattern, they would all give much the same results. This is, however, believed to be an over-. simplification,. and indeed is only a rough approx1mation. Compared directly, there is little similarity between the results. For example, the large Landolt C and the small bar pattern require the same contrast threshold for resolution at a background level of ·-1·80 Log mL. Below this adaptation level, the threshold of the large Landolt C becomes progressively lower relative to that of the small bar pattern, while above this level, the reverse is true. Also for the highest background level, a contrast can be found for which all the bar patterns but none of the Landolt C's can be resolved, whereas at the lowest background ~uminances a contrast level which just allows all the bars patterns to be resolved, only leaves the small Landolt C unresolved. This again shows that the bar pattern becomes more visible in relation to this 'white' Landolt C if the background level is raised, at least for the range of background luminances used here. It will be suggested later that this is because the Landolt C is resolved using primary recipient units, while the bar pattern is resolved by line/edge detectors.

In Figure 6, the increment thresholds for the resolution of the 'double disc' and bar patterns are compared. The comparison shows some fea tures similar to that between the Landolt C's and parallel bars. The double disc in gener al has a hi gher threshold, and the threshold increases relative to the bar pattern as the background level is raised. At the lowes t background levels the thresholds for the double disc are much closer to the thresholds of the parallel bars than to those of the Landolt C. Thus, i f we attribute the behaviour of the double disc to primary recipient units and that of the bar pat~erns to line/edge detectors, we must also expla1n. the difference between Landolt C and double d1sc.

The increment thresholds for the resolution of the Landolt C and the detection of the squares are compared in Figure 7. For background luminances of lo-3 mL and above, the two sets of curves follow each other very closely. This suggests that in this region, resolution of the gap in these white Landolt C's is determined.by the detectability of a square decrement of l1ght equal in area to the gap, viewed against a similar background. This presupposes that increment and decrement thresholds for detection of a square are very similar. Below lo-3 mL, the increment thresholds for the squares fall progressively below those of the Landolt C. The curves for the squares are the steepest of all the patterns at the very low background levels.

In Figure 8, the increment thresholds for the resolution of the broken line are compared with those for the detection of the square. For background luminances above 10- 4 mL, the curves are sufficiently close to suggest that the resolution of the broken line under these con­ ditions is limited by the detectability of a square patch of light of side equal to the dis­ placement between the top and bottom halves of the line. An interesting experiment might therefore be to compare the limit of vernier acuity as it is normally determined with the size of the smallest detectable black square, for a wide range of background luminances.

For background luminances above 10-3 mL, it will be seen from Table II that the increment thresholds for resolution of the medium double disc pattern and detection of the large square are very close. The same is true for the small double disc and medium sized square at background luminances above 10-2 mL.

During the experiment, the subjects were asked whether they could still detect the presence of a pattern when its increment in luminance had been lowered so that it could no longer be resolved. Their verbal reports indi­ cate that at high background levels, they could at best only detect the patterns very faintly in this non-resolving condition. Particularly with the large and medium sizes of pattern, resolution came with detection. At the very low background luminances, most of the large patterns were still only faintly detectable, but the large Landolt C was bright enough to be readily detectable.

The medium sizes of pattern were all detectable, and the small sizes were indeed compared with the background, readily very bright and yet not

resolved.

Extension of the Detection Model to Resolution

Thus far, the results have only been discussed in qualitative terms. In this section, a more quantitative approach is taken by extending the model of detection referred to earlier, to cover the calculation of resolution thresholds. Theoretical curves of increment threshold for resolution, fiiR, against background luminance, I, are derived for the parallel bars, Landolt C and 'discs' patterns, and the effect of different assumptions is examined.

For detection of an increment in luminance fii, on a background luminance I the response of a detector optimally positioned with respect to the stimulus so as to sample I+fii, was compared with the response of a detector sampling the backgroun~ I, alone. It was argued that the difference between the means of these two responses was the 'signal', and that this signal must exceed the combined fluctuations in these two responses by a constant ratio in order for the signal to be detected.

For resolution, it is proposed that the responses to be compared should be from two identical detectors, X andY, which are centred over different regions of the resolution pattern. For the parallel bar pattern, one is centred over a bar, and the other is centred over an adjacent space between the bars. For the Landolt C, one is centred over the gap, and the other over a segment of the ring. In the case of the double disc pattern, one is centred on one of the discs, the other over or near the gap between the two discs. Again, the difference between the mean responses of the two detectors is taken as the 'signal' and it is argued that this must exceed the combined fluctuations of the two responses by a constant ratio in order for the pattern to be

resolved.

Predicted Resolution of Parallel Bar Pattern

For reasons discussed in the last section, the resolution of the bars in the parallel bar patterns will be assumed to be performed by line detectors. Figure 9(a) represents a line detector, X, centred on a bar and a line detector, Y, centred on an adjacent space between bars. The response of each detector to each point of the bar pattern is proportional to the product of the light intensity and the sensitivity of the detector at that point. Thus the effe7tive a~eas, ~E(X) and_~E(Y) of the bar pattern ~n caus~ng a response ~n the detectors X and Y respectively, may be found by integration. The 'signal' now becomes

where C factors light. is a constant embodying geometrical and the number of quanta per unit of The integrated background which con­

tributes to noise becomes

where I 0 representing an "eigengrau" is unimportant except near absolute threshold. Thus from equation (2), by putting Ck = K'

At all but the lowest background luminances, the terms in I 0 and fiiR on the right hand side of this equation are negligible.

The values of rm, ~· T and ~ were first all considered to be set by the backgrgund luminance alone. The theoretical curves thus derived are compared with the empirical data in Figure 10. The values of K' and Io used were -1·22 and 7xlo- 7 mL respectively. It will be seen that the theoretical curves fit the empirical points well for the whole range of backgrounds. However, this good fit is thought to be mis­ leading for reasons explained later.

In the section 'Results', it was noted that at low background luminances, the small bar pattern had to be very bright compared with the surrounding background, for it to be resolved. Therefore, in Figure 11, the theoretically fitted curves have been calculated assuming rm, ~. T and ~pare set by (I+fii/2), the mean luminance within the bar pattern. The fit for the medium and large patterns is hardly affected, but for the small pattern at low background luminances, the theoretical prediction is slightly optimistic.

In Table III, the absolute threshold values calculated from equation (4) assuming rm, ~. T and ~p are set by fii/2 are compared with the measured values. Again, the fit is good for the large and medium patterns, and the prediction a little optimistic for the small bar pattern at low background luminances. This will be referred

to later.

Resolution of the Landolt C Pattern

For reasons discussed in the last section, we shall take it that the position of the gap in the Landolt C is discovered by light integrated in primary recipient units with radial symmetry as in equation (3). Figure 9(b) indicates the positions of the centres of the two units whose responses are compared in order to resolve the pattern. Unit Y was centred over the gap of the C. In the first case, unit X was considered to be diagonally opposite Y, since this would give the ma~imum difference in response. The effective areas aE(X) and aE(Y) of the pattern causing responses in the units X and Y respec­ tivel y were calculated by numerical integration.

By adapting equation (4), the increment threshold for resolution, li~R· is given by

At all but the lowest background levels, the terms in In and liiR on the right hand side of the equation are again negligible.

Figure 12 compares the empirical data with the theoretical curves assuming that rm, ~· T and ap are functions of I alone or of (I+lii). The values of K and In used were -1·22 log units and ] xlo- 7 mL respectively. For both sets of assumptions, the theory predicts thresholds con­ siderably lower than were in fact measured.

It was therefore arbitrarily decided to derive the theoretical curves assuming that the comparison unit, X, was located only one gap's width away from the unit, Y, on the Landolt C ring (see Figure 9(b)). In Figure 13, the resulting theoretical curves are compared with the experimental data. The same values of K and In were used. r , ~· T and ap were considered functions of I onfy. The fit ~s still not good, but is considerably better than that found when the unit X is diagonally opposite Y. The truth probably lies somewhere between, and it may be possible to improve this theory of the resolution threshold of the Landolt C by more refined positioning of the unit X. The unit should perhaps be positioned a fixed distance from Y rather than a distance related to the gap width.

Resolution of Double Disc Pattern

Resolution is again assumed to be performed by primary recipient units. Figure 9(c) represents the two cases considered. In both conditions, unit X was located at the centre of a disc, (see Figure 9). UnitY was located (i) at the centre of the gap between the two discs, or (ii) at a distance of 1·64 times the disc radius from the centres of both discs. The effective areas aE(X) and aE(Y) of the discs in causing a response in the units X and Y respectively, were calculated by numerical integration. Theoretical increment thresholds for resolution were calculated for case (i) and case (ii) by

substitution in equation (4).

Case (ii) provides the larger signal difference ·between the X and Y units and thus predicts the higher sensitivity. In Figure 14, the case (ii) predictions are compared with the empirical data for a value of K of -1·16 log units. The fit is reasonable at background luminances below 10-2 mL, but above this adaptation level, the experimentally determined increment thresholds are lower than those predicted. It is possible that part of the discrepancy at high background luminances may be due to edge detection super­ seding area detection. However, if K had been taken as 1·22, as in Figures 10, 11 and 12, the agreement would have been close at high light levels, but the predicted thresholds would have been 0·6 log units too low at low light levels. A possible reason for this will be discussed later.

The theoretical values derived for case (i) with the unit Y positioned centrally between the two discs have not been plotted because the results showed no improvement.

Detection of Squares

Increment thresholds for the area detection of squares were calculated using equation (5), simplifying integration by assuming that a square is as detectable as an equal area disc for this type of detection. It can be shown numerically that the error in the calculated increment threshold in making this assumption is less than 0·5%. The theoretical increment thresholds thus derived are compared with those obtained experi­ mentally in Figure 15. The values of K and In used were -0·98 log units and 7xlo- 7 respectively. The former was chosen to give the best approxi­ mation at low light levels and for background luminances below 10- 2 mL the predicted values are in good agreement with the empirical measurements, but above this background level we find that the predicted thresholds are about 0·25 log units higher than those measured. To test whether or not this discrepancy is due to edge detection predominating at these light levels, theoretical edge detection curves were derived using equation (4), with the value of K' made equal to that of K used in the case of area detection. The theoretical area detection. The theoretical area and edge detection curves for the squares are shown in Figure 16. It will be seen that edge detection only supersedes area detection for the large square at the highest background luminance. Thus, the simple edge detection so far developed cannot explain the discrepancies at high background luminances. Summation of the four border effects is a possible factor.

A feature of the theoretical curve fitting to the data of this experiment is that the value of K which has been chosen for the squares (-0·98 log units) is considerably lower than the value of K' used for the resolution of the bar pattern (-1·22 log units). Indeed, if the value K = -1·22 log units had been taken, the fit would have been good at high levels of background luminance, at the expense of the fit at the lowest light level, which would then be optimistic by 0·24 log units.

General Comments

We have seen that depending on the choice of K, it is possible to fit theoretical curves either at the upper end or at the lower end of the back­ ground luminance range covered in these experiments.

If K is chosen for a good fit at the lower end, the theory is a little pessimistic at the upper end . If K is taken as -1·22 log units for the best compromise at the upper end, the theory is slightly optimistic at the lower end.

The discrepancy of up to 0·25 log units is in fact a very small one, considering that we are concerned with a range of five orders of back­ ground luminance, but it is, nevertheless, of interest to consider possible reasons for it. There is firstly the possibility that the inherent sensitivity or photon detecting efficiency changes with light levels. There is in fact a change from the use of rods at the lower end of the range used in these experiments, to the predominant use of cones at the upper end. This could readily alter the balance between the receptor sensitivity at the lower and upper ends of the range, by the small amount in question.

Secondly, there is some evidence that the inverse cubic law adopted for convenience in describing the distribution of sensitivity across a primary recipient unit may be unduly sharp in the centre. Moreover, the use of the same law to describe the transverse variation of sensitivity across each component of a boundary detector may also be unjustified, because almost any means by which the outputs from primary recipient units were combined to form linear detectors would introduce at least some rounding of the peak.

The data which originally justified the use of the inverse cubic law is not very sensitive to the sharpness of the central peak, but resolution threshold data, such as that for parallel bars, is very sensitive to the peak when the width of the bar is small compared with rm. This could account for . the discrepancy between the pre.;!icted and observed thresholds for the small parallel bar patterns at low light levels in Figure 11. The modification to the peak of the inverse cubic curve necessary to allow for this is indicated by the dotted curve in Figure 1. Some comparisons · have also been made with published data on contrast sensitivity functions6 which support the idea of a slightly more rounded peak than the pure inverse cubic curve employed, and the question is being investigated further. At high light levels the rounding may be partly due to the influence of the optical system of the eye5.

Thirdly, there is the question of boundary detection taking over from the radially symmetrical primary recipient units. This has been considered earlier in relation to the square pattern, but there is some uncertainty regarding the relative sensitivity of the two processes and this can influence the balance between recognition thresholds at the two ends of the scale.

Discussion

The outstanding fact about the results reported here is that, within the limits of the slight discrepancies which have just been discussed, it is possible to work on the assumption of such a uniform sensitivity of photon detection. Thus, once one has recognised the limitations of temporal and spatial summation in the retina, the efficiency of utilisation of the information in the incoming photon flux appears to be very nearly uniform over a very wide range of light levels. Considering the variety of visual tasks included in these experiments the uniformity of behaviour is surprising. Thus the theory gives some insight into the reasons for the characteristic differences in observed thresholds between the different forms of resolution test pattern.

The theory and its application to the experi­ mental data is aimed at discovering how efficiently the eye uses the light in the retinal image. It has not primarily been concerned with the detailed nature of the mechanism by which the eye performs the operation. Thus, there is no attempt to identify the cause of the variation of rm, Lm· etc. as a function of light level. Nor has there been any attempt to explain what neural mechanism would provide such uniform efficiency of extraction of information from the incoming light. This is an interesting field for con­ jecture, but there are many alternative possi­ bilities to choose from, and a proper discussion of these must await a later occasion.

It is encouraging to find in the need to invoke line/edge or 'boundary' detectors some measure of correspondence with recent physio­ logical findings 7 · For the present, whatever may be the biological significance of the summa­ tion areas and edge detectors, their practical significance as parameters determining the performance of the eye is a real one. They indicate clearly the nature of the limitations under which the eye operates when detecting patterns under conditions of poor visibility. Moreover, the concept of a photon noise limited threshold is supported by further experiments reported separately8 in which artificial noise, deliberately introduced into retinal images, appears to have an analogous effect on detection.

The work reported here has been on relatively simple patterns and, perhaps in con­ sequence, fairly simple detecting mechanisms have come to light. The situation may well be complicated when one considers more complex patterns, especially if these are seen against an inhomogenious background. It may well be that there exist more specialised detectors for particular patterns or features of patterns. Indeed, recent physiological evidence suggests there may be, but their existence cannot be deduced from these experiments. Then again, both types of detector discussed here are taken to represent the mean performance averaged over all the detectors which take part in the recog­ nition of different patterns on different occasions. It is, however, clear from physio­ logical evidence that there is appreciable variation between the response of different ganglion cells and this may make the concept of an "average" detector shape of less value when more complex patterns are considered. The problem may resolve itself round the question of connectivity between less and more complex detectors. If the more complex detectors are formed by combining the outputs if simpler detectors then the former may be expected to be subject to the inherent limitations of the latter. If, on the other hand, more complex detectors can obtain their information directly from receptor cells, these limitations may not apply. The facts will only become apparent with further work.