The Effect of Noise on Visual Pattern Recognition

Introduction

It was first suggested by Barnes and Czerny that statistical fluctuations in the arrival of photons might present a limit to the performance of the eye at low light levels. The theory was 1 put forward more ~ormally almost simultaneously by two workers, Rose and de Vries3, but the full statistical approach was not proposed until the publication of a now famous paper by Rose4 in which the actual performance of the eye was compared with the highest performance attainable with an ideal detector limited to the physical dimensions of the eye (principally the pupil area).

Rose originally conceived the ideal detector as detecting a faint pattern superimposed on ~ uniformly illuminated background. The basis of Rose's theory is that, if an ideal detector is continually taking samples of a mean number, n, of events from a display of individual, equal energy, events, then an incremental increase 6n in the number of events in a sample will be required for the presence of the increment to be detected with a predetermined certainty; the magnitude of the increment being given by the equation:-

A Pois·son distribution for the detected photons is assumed ·in representing the magnitude of the noise present in a sample as ~. and k is a constant, named by Rose the limiting signal to noise ratio of the device. The symbol k has also been referred to as a factor of certainty or reliability, since the reliability of a decision will, of course, depend on the signal to noise ratio of the data on which it is based,

The ideal detector and the visual system

Rose's ideal detector 4 took samples of the same shape and size as the pattern to be detected, the sensitivity of the detector being uniform over the whole of the area of the sample. Detection of a pattern against a background would involve com­ parative measurements within the pattern area and of the background alone, the difference giving the signal which the eye utilises, This signal is thought of as being detected against the statist­ ical photon noise in the background picked up by both detectors, This is the noise which unavoid­ ably arises from the quantum nature of the detect­ ion of light, and should, of course, include the quantum noise in the signal too, if this is significant in comparison with the background,

The visual system, on the other hand, appears to be constrained to use samples of light falling within certain summation areas, This question has been discussed in some detail in two other papers?,3 It appears likely that the eye takes samples either within an area of radial symmetry, the dimensions of which depend on the mean light level (a primary or area detection), or within an elongated area (or linear detector), The eye also appears to be constrained to integrate over a period of time determined by the mean luminance of the local background.

We may calculate the signal and background contribution by integrating the light falling within two such summation areas strategically placed with respect to the retinal image of the pattern to be detected. Thus, the difference is a measure of the number of photons available to form the signal 6n by which the presence of the pattern may be detected. Similarly the sum is a measure of the total number, n + 6n of photons in the background and signal detected within these

summation areas,

We may now make use of Roses basic which expresses the need for the signal the statistical noise in the background reliability factor, k, viz. . 4 equat~on to exceed by a

The insertion of 6n in the noise term is only of importance near absolute threshold. For the majority of experiments reported here it has been possible to omit it as in Rose's original form (equation (1)). On a basis of c~lculations from this equation, it has been found that a reason­ able fit to the empirical visibility of discs of various sizes and contrasts is obtained if the sensitivity within a primary detection is weighted according to the expression where r is a function of the mean luminance of the backgro~nd as shown in Figure 1, and both r and rm are measured for convenience as angular dis­ tances subtended at the centre of the lens.

The effect of a background illumination, I lumen per cm 2 incident over the whole extent ofra summation area on the retina would then be proportional to:-

where S is the sensitivity of the eye, expressed as its ~uantum efficiency.

and f is the focal length of the lens in ems.

The effect on the summing unit is seen to be the same as that of a disc stimulus of radius r on a detector of uniform sensitivity S , m 0

We can now proceed by determining the number of quanta arriving in the samples of stimulus and background, and so find the contrast required for detection in the presence of the noise caused by the fluctuations in these small numbers of quanta,

The sample size of the visual system

The number of quanta detected within the summation time T of the eye will be

where P is the number of quanta per second per photopic Lumen. Taking account of the geometrical factors in the formation of the retinal image, this may be re-expressed in terms of I, which is the luminance in Lamberts of the background presented to the ~ye, thus the mean number of quanta in a sample of the background will be:

where a is the area of the pupil of the eye, and A is eqflal to nr 2 and represents the effective size of the summWtion area of the eye in units of so lid angle,

The mean number of quanta constituting the signal sample requires the evaluation of the integral of the sensitivity curve of the summation area over the extent of the stimulus. To illust­ rate this calculation we shall here consider a stimulus in the form of a disc, radius R, in which case the mean number of quanta in a signal sample may be ~ound from the equation From which, arguing as before, we may express lin in terms of the luminance I'll presented to the eye:

By substituting n and lin in Rose's equation (1), we have at threshold:-

If we define threshold contrast required for detection as C = lll/1, we have

We wish to fit this equation to our experi­ mental results, the term k being determined by rs

the actual positioning of 0 the curves for the best fit. However, before we can do this, we must evaluate T, the summation time of the eye, To establish the value of T, it has been necessary to resort to the literature in papers by Barlow? and by Graham and Kemp,B Their results are shown in Figure 2, in which the summation times plotted are those of the limit of full summation.

Theoretical area detection and empirical results

Comparison between theoretical predictions based on equation (7) and empirical measurements of thrgsgold, have been given in detail else­ where. ' For patterns such as parallel bar resolution patterns and annuli, equation (7) must be replaced by a similar equation based on the concept of 6 elongated "linear" or "edge" detectors, These patterns will not concern us in the experiments reported here,but it does appear that the concept of a "linear" or "edge" detector is probably relevant, at least for large discs at high light levels.

For the detection of discs, it is optimal for ~~e:~~=i~~s~r~~t=~~yt~~c~~:~st~h~es~~~~=~ic, area,

Sample curves for theoretical predictions fitted to empirical data from reference 5 are repeated here, in Figure 3, for discs alone, In fitting the curves to the empirical data, only a movement of all the curves together in a vertical direction is permitted by adjustment of the value assumed for the ratio of constant~ appearing 0

in equation (7), From the final best fit obtained 0 ~ can be evaluated, It will be noted ~so

that at higher light levels 0 the full line theoretical curves are based on the concept

. 5,6 of line or edge detect~on,

It is as though the two forms of detection work in parallel, their output summing as in an 'or' gate. Whichever is the more sensitive auto­ matically takes over the task of detection; thus, the curves of Figure 4 are compound curves, The contrast required for detection by each process has been calculated, and the minimum of the two plotted,

It is interesting that this theory allows us to assume both k and S relatively constant over a wide range of light 0 levels.

Using the values:-

) -1 -1 and p = 2.52 x lo1 S quanta (507 m~ sec lumen , a value of k of 2.1 was established. (Or possibly rso

up to 6, depending on the assumptions we make about the nature of the detection process),

Experiments with noisy patterns

The work which has just been summarised is based on the concept of a detection process limited by quantum noise. The justification for this con­ cept is the relatively close agreement between the theoretical predictions based on it, and the empirical data on visual thresholds. It would, however be of interest to have independent evid­ ence of 0 the influence of noise on the detection process, and for this reason experiments were planned in which dynamic noise was deliberately added to the patterns presented to the eye, Modification to the theory to take this additional noise into account will be described, and the predictions of this theory will be compared with observation of the visual threshold for noisy patterns of a number of observers,

The form of noise chosen for these experi­ ments was quantum noise of the type one might expect in the output from an image intensifier. This noise arises owing to the detection of individual photons at its input photo-cathode, This form of noise has the advantage that it is essentially the same in nature as the quantum noise we have assumed to limit detection in the eye, though the size of the "quantum" presented to the retina is different. There is the added con­ venience that calculation is straightforward, albeit somewhat laborious for certain patterns.

An extension of the detection theory

The fall off in sensitivity towards the edges of the summation area can be regarded either

• (i) as a lowering of the probability that the arrival of a photon would be recorded by the summing unit, or
• (ii) as a diminution of the effect of each photon at the summing unit,

The distinction has hitherto been relatively unimportant because it only makes a difference to a single multiplying constant common to all the predictions, By integrating as in equation (3), and using the results in equations (4) and (7) to represent mean square noise, it has implicitly been assumed that the former assumption is just­ ified. The latter assumption could have led to a reduction in calculated mean square noise of up to ten times. Thus in making comparisons between predictions and theory in order to est~mate :he numerical value of k , the value of th~s rat~o !So could be as high as 6 or 7, From now on, to emphasize the importance of this difference, the effective area over which the background is integrated to determine the mean square noise will be denoted as AS instead of A. If we adopt assumption (i), then S = 1, but it will be shown that s can be lower than this according to assump­ tion (ii), We now wish to consider the integrated effect within a summation area of a display made up of a large number of spots of light. We shall assume that the effect of a spot of light will decrease as its distance from the centre of the area increases, according to the law previously assumed for S , in equation (1). r The noise due to photon detection in the ret­ ina remains a function of mean retinal 'illumin­ ation as before. The noise due to the presence of individual spots may be calculated as follows.

The effect of visual noise on the summation area

If we consider the receptive field to be stimulated by a random display of N spots of light or "dots" per square centimetre per second, then the mean number of spots of light falling on an annulus of width or and radius r in time T is:-

where v is t .he distance of the obsetver £rom the display in centimetres and r and 6 r are angular distances subtended at the centre of the lens. The mean square variation in this number is also

Now, the mean number of photons detected from each dot will be:-

where Q is the light energy of a dot ~t the display.

a is the area of the pupil in square p centimetres,

f(r) is the function describing the sensitivity curve of the summation area,

p is the number of photons per unit of light energy, i.e. per Lumen Sec,

s is the quantum efficiency of the eye. 0

The variation in the number of dots in an annulus will, then, give rise to an additional mean square noise whose magnitude corresponds to the following mean square fluctuation of detected photons:-

The effect on the whole of the summation area will be the integral of the above expression over the area:-

A' ·············· , , ··· , ·· (9)

00

J2nr.f2(r).dr

where A'=

0

This additional mean square noise must be added to the mean square noise due to photon fluctuation in the background light level N .Q presented to the eye, The expression for the number of quanta detected in a sample of back­ ground illuminated by N spots/cm2sec. can thus be written, from equation (4)

For the reasons given before, in using this as an estimate of the mean square noise, A will be replaced by AS, givtng

If we now assume that a signal is represented by a small positive increment, ~N, in the density of dots in the display, we can, by summing these two noise terms, represent Rose's equation by:-

where a 9 is the effective area of the stimulus; for a d1sc, radius R, this would be n(l/R+l/r )-2 as derived in expression (5). m

Equation (10) can be made more manageable by writing

so that S represents the sensitivity of the eye in terms of detected quanta per unit of luminance entering the pupil, Equation (10) then becomes:-

which can be written:-

A' can also be evaluated:-

The general expression for detection in the noisy display becomes:-

If we adopt assumption (i) referred to above, AB arises from the integral in equation (3) and B = 1. If, however, assumption (ii) is adopted, it becomes appropriate to use the integral in equation (12) and AS = A/10.

The representation of temporal and spatial sununation in terms of llght level

In Figure 2 a straight line was fitted to the data of Barlow7 and Graham and Kemp8 for the summation time of the eye, It is

The fit is seen to be good, and we can thus substitute forT in equation (13),

In Figure 1, a straight line is fitted to the empirical points as shown, The equation is

The fit is not as good as the line representing summation time, but is acceptable as an approximation.

If N Q = I, then the summation time T, summation area A, and its radius rm' can be represented algebraically by equat1ons such as:-

where k 1 and k 2 , and x and z derive from the empirical data qu~ted. Substituting these expressions for T and A in equation (13) gives:-

the value of J:. being given by:- ao

From the general equation (17), have been ealculated curves of contrast versus light level for various sizes of disc stimuli (see Figure 4). The effective area of the stimulus a , has also been calculated for rectangles by n~erical integration, and Figure 5 shows the contrast ver­ sus light level curves for four sizes of rectangle, their areas being the same as for the discs in Figure 4.

It was assumed, in ealculating the curves of Figures 4 and 5, that the light level was varied by changing Q &ld keeping N constant, This is the way in which the output display of an image intensifier would vary ·if the photon gain was varied, with a fixed illumination on the photo cathode. It is also the way in which the simulated image intensifier display altered as the subject controlled the brightness of the display in empirieal experiments.

It will be noted that the stimuli, instead of gradually becoming ntore detectable as the light level is increased in this way, should be optim­ ally visible at a particular light level, and thereafter become less visible.

A simplified explanation of the theory

Mathematical equations are not always the best medium for imparting theoretical concepts, and so it is felt that a brief stwnary of the above theory would be advantageous.

The basis of the theory is Rose's equation:-

h · h ·J.· f h C · ~n b · w J.c , t e contrast, , J.S -, can e wrJ.tten:- ii

where .!. is a measure of the amount of noise vi

present in a sample. This indicates the need for the mean number of ·observed events n to be sufficient for one to detect the requisite contrast.

We need now to identify the events which must exceed a certain mean number for the pattern to be visible. The answer to this is simple, If there are plenty of light quanta per dot, but few dots available to denote the presence of the pattern, it is the number of dots which is important. If, on the other hand, there are plenty of dots, but very f ew quanta per dot, it is the number of quanta which control the visibility of the pattern. Now, which of these two situa­ tions arises will depend on the light level.

and

1 . . . 1 I-0.5 the samp e sJ.ze A T J.S proportJ.ona to

This means that if the ambient light level is decreased by a factor of 100, the sample size taken by the eye increases by a factor of 10. Now the significant number of events in a sample can be either the number of quanta or the number of dots, whichever is the smaller. Thus, at high light levels (high values of Q), we have a small summation area containing a small num~er of dots but a large number of quanta per dot, as shown in the diagram at the bottom right hand corner of Figure 6. Under these circumstances the number of events (or dots) decreases as the light level is increased, because both the summation area A, and the summation time T, decrease, The pattern therefore becomes less visible with more light.

At threshold at low light levels (low values of Q) there are few quanta per dot, although the size of the sumr.1ation area has increased and will probably encompass more dots,as shown on the left hand side of Figure 6. It is now the number of quanta that control visibility and thus visibility improves with increasing light level, the relatively low rate of change of summation area and time being more than balanced by the change in light quanta available.

The final result is that visibility is optimum at one light level and drops with either an increase or decrease in light level. Thus, we have a dip in the curve of Log contrast vs log light level as indicated in Figure 6, and shown in more detail in Figures 4 and 5, The relative steepmess of curve to be expected on either side of the minimum depends, of course, not only on the rapidity of change of T and rm, but also on the size of the pattern relative to the summation area. If the pattern is smaller than the summa­ tion area, then variation of summation area with light level will have little effect on ~n and the rising slope to the right of the minimum will be much less steep.

A special case

When the pattern is large compared with the summation area, i.e. R >> rm' equation (18) simplifies to:-

and equation (17) becomes:-

As Q is varied, the value of C defined by this equation has a minimum when the terms within the root sign have a minimum. The minimum of

occurs when

or

but, if x·= 0.4 and z = 0.1, as we have postulated, then:-

for a minimum.

If the minimum of such a curve is establishe~ we can determine a value of S, and thus S0 , for either assumption regarding e. This gives S0 independently from k and hence separates the terms in the lumped constant, k referred to earlier. rso

A preliminary empirical investigation of the detectability of patterns immersed in noise

When the existence of a predicted optimum light level for visibility was encountered, experiments were designed to explore the region of the minimum required contrast. Noisy images of various degrees of contrast were recorded on cine film, so they could be presented to observers. They were told that there might be an optimum light level and were instructed to determine the lower threshold of visibility and also a higher threshold, if one should be apparent. If no optimum could be found, or no pattern could be seen, they were asked to report accordingly. If they thought they could see a stimulus at a particular light level, and not above or below this light level, they were instructed to set the display to this light level and inform the experimenter that they had found an optimum.

Results

The thresholds set by two subjects are given in Table I. Qualitatively, the results confirm the existence of an optimum light level. The higher threshold has been determined in most cases where the theoretical predictions led to the expectation of a minimum in the curve of contrast vs light level within the range of light l evels covered.

Analysis of the Results

The anlysis of the results may be performed by comparing empirical upper and lower thresholds with theoretical predictions. Figure 7 shows theoretical predictions based on the measured number N, of dots per unit area in the film. The position of the curves to the left of the minimum depends principally on the ratio k~, which has been taken as 2.26 in accordance w~th the mean measured over previous experiments for observer J.E. The position of the curves on the right hand side depends principally on k. Thus, for k~ fixed, there are several curves for different values of k.

The measured contrast of the film is indic­ ated by the dotted line. Now if, for example, we assume that k = 1, the predicted upper and lower thresholds would be expected where the curve for this value of k cuts the dotted line. These points are indicated by crosses. In fact, the measured upper and lower thresholds for J.E. are as shown on the horizontal axis. These do not coincide precisely with the predicted values, and by working back one can deduce modified values of k and S 0 , which correspond to the empirical measurements of threshold. For covenience, in comparing the empirical results all the measured thresholds have been converted in this way to equivalent 'empirical' values of k and S/ 15 . These are shown in Table IIa.

Roughly speaking, it is the position of the minimum on the horizontal axis that determines S, as indicated by equation (20). This relation­ sRip is exact for large discs, but the inter­ dependence is more complex for smaller discs .and other shapes. As k is varied, for a given S 0 , there is a proportionate vertical displacement of the predicted curve. Thus the horizontal separation of the points of intersection with the dotted 'contrast' line depends on k and S , roughly speaking, it is the horizontal se~aration of the two measured thresholds that determines the 'empirical' values of k. The relationship is again only simple for large discs and can involve appreciable computation for small discs and other patterns, especially when the slope differs either side of the minimum.

It will be evident that if measurements are made in a region where the theo~etical curves shown in Figure 7 have a small gradient, the observer might be expected to have some difficulty in determining the threshold accurately. The repeatability of threshold measurements was within about 0.3 log units, rather poorer than most of the measurements made without noise present. This might be expected to lead to errors in S 0 /15, of perhaps 0.5 log units in magnitude. Large errors might be expected where the observer has attempted to determine the light level for optimum visibil­ ity. This must be borne in mind in looking at the values derived in Table IIb, all of which were derived from a single measurement of optimum light level.

Conclusions

The mean value of k in Table IIa for observer J,E. is 1,28, with a mean square deviation of 0,33. For observer J.B, the mean is 0,63, with a mean square deviation of 0.33. These results suggest a significant difference between these two observers, and imply that it is necessary to treat observers as individuals in determining this parameter.

The mean value of S /8 for J,E. is 0.485, with a mean square devia~ion of 0,52, and for J,B, is 0.078, with a mean square deviation of 0,069, The percentage variation here is much larger, but again there is a hint that the differences may be significant and that observers shculd be treated as individuals.

The interesting feature of these results is that when we consider the quotient

which is the parameter which is important in nor­ mal vision, the difference between the observers appears much less, J.E. giving 2,17 and J,B, 2.91, with mean square deviations of 0.75 and 1,73. These may be compared with the value quoted previously, of

which was the mean of a large number of measure­ ments of normal vision at low light levels. In these earlier measurements the factor 8 was assumed to be unity, in accordance with the conventional application of Rose's equation, so this value is directly comparable with the present results for

Now that it is possible to separate k on the one hand and S /8 on the other, it becomes of great interest 0 to know what assumption to make about 8. If one argues that the method of inte­ gration within a summation area is likely to have evolved in such a way as to maximise the sensi­ tivity of the eye, one is arguing in favour of assumption (ii) discussed earlier. This assumption leads to the minimum contribution of competing noise for a given rate of detection of photons within the boundaries of a summation area, as indicated by taking 8 = 0.1. If this assumption is correct, the results would give us efficiencies of detection S 0 of 4.8% for J,E, and 0.78% for J,B,, which is reasonably in line with attempts which have been made to estimate this from other data.

These results, as we have already said, represent a preliminary trial of this method of measurement. The tentative conclusions clearly justify continuation of further empirical work, to confirm the findings and to investigate factors such as the variation of k and S 0 /8 between individuals, When computing the results of a more comprehensive experiment, it would be desirable to introduce pupil area as a variable, and perhaps to employ a modified weighting function, Sr, to des­ cribe the variation of sensitivity across the sunnnation area. 6