Towards Understanding Color Ocular Fundus Images

1. INTRODUCTION

Color ocular fundus images are used in the mass diagnosis of adult diseases such as hypertension and diabetes. Since the photo­ graphing system of them is simple and cheap, if the automatic understanding system of them is realized, the mass health management by them together with chest X-ray images will be much more popularized. There are a few exam­ ples of computer analysis of ocular fundus images [ 1] , [ 2] , [ 3], [ 4] , [ 5] .

For color ocular · fundus photographs, as far as we know, there has been only one exam­ ple of computer analysis. It is the pattern recognition of color fundus images by Yokouchi et al. [4], [S]. They investigated the possi­ bility of automatic diagnosis of arterio-venous crossing phenomena. But their method is inter­ active in the sense that they indicate the cross points manually before the computer analysis. The change of venous caliber before and after its crossing was measured along its course and the minimum value of each normalized caliber measurement was utilized to grade the crossing signs. The researches mentioned above stay in the phase of analysis. The ob­ jective of our research is the understanding

This work has been supported by Pattern Information Processing System Project of the Agency of Industrial Science . and Technology of the Ministry of International Trade and Industry of Japan. ·

of color ocular fundus images by computer, which is inevitable for the automatic mass diagnosis of hypertension and diabetes. The theme in this paper is to locate blood vessels and iden­ tify arteries and vains automatically. This process is fundamental for the analyses of not only arterio-venous crossing phenomena but also other symptoms of adult diseases [9].

We have to make clear the levels of knowl­ edge [10] that can be used in each stage of the process.

For the first step we analyze the chromatic characteristics of color ocular fundus images. We get the signal level knowledge on them which is utilized in the stages of extracting blood vessels and most reliable labeling of initial line segments.

We propose a dynamic thresholding scheme for the extraction of blood vessels. It is very effective against images which have shad­ ings or unevenness of light intensities. Thirdly a deterministic labeling algorithm of blood vessels is proposed in which the physical level knowledge on them is implemented. The lables of some vessel segments from which the labeling starts are determined by the signal level knowledge and some positional con­

straints.

2. CHARACTERISTICS OF COLOR FUNDUS IMAGES

The first step of understanding color ocular fundus images is the analysis of their photographic and chromatic properties. They are taken on the 35 mm color reversal films by the fundus camera. It requires some skill to take the fundus photos of good quality. Since the state of fundus is individually different, they have some variation of color tone. More specifically, there are two main causes of the variation--the photographic system and the individual. The former involves the spectral characteristics of color films, the condition of photo processing, the optical property of the fundus camera and the amount of light. It is comparatively easy to prescribe these con­ ditions and they do not have so much influence on changing color tone. The latter comes from the age, the size of pupil and other conditions of a person [1]. It changes color tone of photos remarkably. Therefore, it is very im­ portant to select parameters that are not influenced by each photograph, in other words we should use relative (not absolute) color information.

2-1. Digitization of Film Images

In our experiment we Hsed a high-speed drum scanner which had the Wratten No.25, No. 47B, and No.58 filters for tricolor decomposi­ tion. The photo density was quantized to 256 levels.

The color fundus images on films were scanned by the drum scanner with the sampling pitch of 50 ~m, and di~itiz~d to 255 x 256 pixels (i.e. 12.8mm x 12.8mm on film).

2-2. Chromatic Analysis of Digitized Fundus Images

A fundus image consists of four portions; Retina, Disc, Artery, and Vein. We plotted the chromatic information of the digitized data in the sample areas of each portion onto the UCS color coordinate system (u, v, V).

The distribution in u-v plane shows that it is parallel to the u-axis, which means the chromatic information of this sample fundus image is approximately represented by the u­ coordinate. Furthermore the distribution of retina is roughly separated from that of blood vessels. But the distributions of artery and vein can't be distinguished.

From the distribution of t .hose data in

V-u coordinates we see that arteries cant' be completely separated from veins also in this case.

Certainly some confirmed artery or vein pixels are determined by (u, v, V) values.

The analysis tells us that the labeling of blood vessels requires not only signal level knowledge mentioned above, but also physical level knowledge which will be introduced in section 4.

3. A DYNAMIC THRESHOLD SELECTION FOR EXTRACTING BLOOD VESSELS

The extraction or distinction of blood vessels from the retinal background is a mile­ stone of understanding fundus images.

Color ocular fundus images contain an amount of shading or unevenness of light inten­ sity. So the ordinary global threshold selec­ tion methods fail to extract blood vessels. Dynamic threshold selection schemes are desira­ ble in such circumstances. Chow and Kaneko's method [6] is a well-known one which has been applied to the extraction of blood vessels in fluorescence fundus images with some modifica­ tion by some researchers [3] with fairly good results.

The contrasts of fluorescence fundus images are much higher than those of color fundus images. The shortcomings of this method when applied to the extraction of low contrast blood vessels are as follows. The thresholds of all subimages cannot necessarily be obtained and the distribution of them is non-uniform in the image because of the irregular pattern of blood vessels. Therefore, the error of interpolating the threshold at each pixel is often bigger than the contrast between blood vessels and the retinal backgrouttd, resulting in noisy binary images. We abandoned applying Chow and Kaneko's method to color fundus images because of their low contrasts and unevenness of light intensity. We have developed our own method.

3-1. A Dynamic Threshold Selection Scheme

The gray image is splitted into a set of blocks (subimages) as in the case of Chow and Kaneko's method to eliminate the influence of shading. In each block it is examined whether the boundary of blood vessels and the background is included or not by taking the derivative and binarizing it with variable thresholds. When boundaries are detected in a block, the bistro­ gram of pixels only around boundaries is made to determine the threshold for this block. If a block does not include any boundary, It is classified into the background. The size of a block which is given in advance is large enough to contain full width of a vein or an artery. The flow of this scheme is the following.

1. Split the image into N blocks. For each block, do the next steps.
2. Apply the Laplacian operator to the block. The result will be called the Laplacian subimage.
3. Binarize the Laplacian subima~e by varying the threshold and che~k whether there is a connected compo­ nent whose size satisfies some con­ ditions. If not, then set the flag of this block to 0 representing that it belongs to the background. If it exists, the flag is set to 1 showing that this block includes some parts of vessels.
4. {4) For each block whose flag is equal to 1, the histogram of pixels only around boundaries is made to determine the threshold by Otsu's method (7] and the block is binarized. In the binary block the size of each connected com­ ponent is measured and if it is less than a given value, the component is eliminated as a noise.

We should add some words on step (3). When the Laplacian subimage is binarized, the threshold is decremented first from the given upper value until the size of maximum connected component satisfies a certain condition. If any line segment which corresponds to a part of a boundary exists, then the threshold is in­ cremented next from the ·given lower value until the size of maximum connected component satis­ fies another condition. This procedure is im­ portant in order not to miss low contrast boundaries which also exist in the block.

For the conditions mentioned above we can set some constraints on the size and the shape of the extracted boundary in the subimage.

4. LABELING BLOOD VESSELS

The second stage of fundus image under­ standing is the labeling of blood vessels. There are two kind of vessels; arteries and veins. Our aim here is to locate them, which leads to the detection of crossing points and to the measurement of arterio-venous crossing phenomena such as tapering, banking, Salus' sign, S-shaped bend and parallel-Gunn [9].

The physical level knowledge on blood vessels that can be used at this stage is as follows.

• (i) Arteries (Veins) do not intersect each other. In other words arteries (veins) intersect only veins (ar­ teries).
• (ii) Arteries (Veins) branch off only from arteries (veins).

In order to use this knowledge positively we have adopted the following procedure for locating arteries and veins.

1. Make the thinned image from the binary vessel image.
2. Detect characteristic points in the thinned image and make the line segments list.
3. Label each line segment, namely determine whether it is a part of an artery or a vein.

4-1. Skeleton Patterns of Binary Blood Vessels

The state of blood vessels such as branch­ ing, crossing and meandering is simply repre­ sented by that of their medial axises or skeletons. The direct results of thinning binary blood vessel images contain small circles, isolated points and prickles that act as noises. We first eliminate them and start with thinned images. In order to analyze blood vessel networks we detect characteristic points and list up line segments. We define three kinds of characteristic points: end point, branch point and cross point. A line segment is defined as a segment of medial axis whose head and tail are characteristic points. Therefore, there are nine situations of a line segment. Several parameters on a character­ istic point and a line segment are measured and are written into the characteristic points list (CP-LIST) and the line segments list (LS-LIST), respectively. These lists are very useful to activate the physical level knowledge mentioned above.

* Characteristic Points List (CP-LIST)

It includes several parameters of each characteristic point. They are the (x, y) co­ ordinate, the characteristic index which repre­ sents whether the point is an end Pt. or a branch Pt. or a cross Pt., the numbers of the line segments which meet at this point, and so forth.

* Line Segments List (LS-LIST)

The parameters of each line segment in the LS-LIST are the characteristic point numbers of the head Pt. and the tail Pt., the length, the directions of this line segment at the head Pt. and the tail Pt., the chain code and others.

4-2. Labeling Algorithm

The goal of labeling is to decide that each line segment belongs to an artery or a vein or the retinal background. The labeling starts with some line segments that have most reliable initial labels determined by the signal level knowledge.

4-2-1. Initial Labeling

The gray values of arteries are smaller than those of veins in a broad sense. If we limit the area in the V-image and set two thresholds 6L, 6U in the gray levels, we can expect with certainty that the pixels whose gray values are less than 6L belong to arteries and the pixels whose gray values are greater than 6U belong to veins. 6U and 6L may be set­ tled by some clustering approaches. The most reliable line segments with which the labeling starts are determined as follows. On every line segment of the skeleton pattern in the certain limitted area the number of pixels with a certain label (or no label) is counted and divided by the line length. There are three ratios, PA, Pv, and PN which represent the probabilities that the line segment is a part of artery, vein and unknown, respectively. Hence PA + Pv + PN = 1.0. If a line segment has the probability of PA or Pv higher than a given value, it is chosen as one of the most reliable line segments.

4-2-2. Labeling Procedure

The labeling starts from the line segments which have the most reliable initial labels. If a non-labeled line segment is connected to other labeled line segments, it will be labeled as follows using the physical level knowledge mentioned before. The procedure also takes into account the natural directions of blood streams.

Since the full program structure of the procedure is lengthy to be described here, only the essence of it is presented.

At the head point (H.Pt.) and the tail point (T.Pt.) of a non-labeled and non-isolated segment whose angle between I is nearest to 180° and call it LTl. Label(!)= Label(LTl). Exit.

If Label(!) ; 0, increment Conflict Flag(!) by one and exit.

• Label(LTl) ; 0 or Label(LT2) ; 0
• ~ If Label(!) = 0, then let Label(!) = Label(LTl) or Label(LT2), and exit. If Label(!) ; 0 and ; Label(LTl) and ; Label(LT2), increment Conflict Flag (I) by one and exit.
• Otherwise, exit.
• When the T .Pt. is a crossing Pt., let the other three line segments be LTl, LT2, and LT3.

<D Choose the line segment I I whose angle between I T.Pt. 1 I is nearest to 180° and -iTr-7(~, call it LTl. If Label(LTl) ; 0, and Label(!) = 0, then ,' LT3 ' Label(!) = Label(LTl) ,'LTl and exit. If Label(LTl) ; 0 and Label(!) ; 0 and ; Label(LTl), increment ConflictFlag(I) by one and exit. Otherwise do the next steps.

• Label(LT2) = Label(LT3) = artery (vein) ~ If Label(!) = 0, then let Label(!) = vein (artery).

If Label(!)= artery (vein), increment Conflict Flag by one. Exit.

• Label(J.T2) ; 0, Label(LT3) ; 0 and Label(LT2) ; Label(LT3)

~ Increment Conflict Flag(!) by one and exit.

• Label(LT2) =artery (vein) or Label(LT3) = artery (vein)
• ~ If Label(!) = 0, then let Label(!) = vein (artery). If Label(!) ; 0 and Label(!) = artery (vein), increment Conflict Flag by one. Exit.

The labeling procedure terminates when the number of non-labeled line segments does not decrease anymore by iteration.

The isolated line segments which have no reliable initial labels are left to be non­ labeled. They need another analysis. The con­ flicts are caused by artifactitious line segments or by disappearance of true blood vessel line segments. Since the binary blood vessels before thinning have widths at cross points, artifacts are made after thinning. In other words a cross point before thinning is splitted into two branch points after thinning.

Another example of artifacts is a bridge caused by noise. The disappearance of a true vessel line segment has the same effect as a bridge. We need more details of the image in order to distinguish the disappearance from noise. From this analysis we know that the conflict flag gives us very important informa­ tion on crossings, noises and disappearance of true vessel segments.

Another useful physical level knowledge which was not taken into account in our ex­ periment is that there should be no isolated blood vessel pieces ~cept concealments. This tells us that the interpolation of line seg­ ments could be done before labeling.

5. DISCUSSION

Here we consider some additional methods that might be useful for our problems.

1. On the extraction of blood vessels:

Another approach is to apply line detec­ tion or ridge following algorithms which are often used to find linear features in LANDSAT images. We have applied one of them to our case and got some promising results.

1. On the labeling of blood vessels:

The relaxation labeling [8] might be ap­ plied to our case. Since the pixelwise relax­ ation spends a great amount of time, we should use it in the stage of line segment labeling. We have to define neighbor relationships of line segments, compatibility functions, and weights. The arithmetic updating rule may be used.

3) From the view point of diagnosis:

In case of the analysis of arterial-venous crossing phenomena, some specified retinal areas are inspected. The area around the fovea which includes only very thin vessels and the area within some radius of the disc are out­ side the inspection. This will go a long way in overcoming the difficulty of noise elimina­ tion.

On the other hand there are important phenomena on arteries which might be neglected or erased by careless noise elimination. They are light streaks on arteries (copper wire arteries) which serves much for the diagnosis of arteriosclerosis but often acts as obstacles to labeling. This problem was not dealt with in our experiment.

6. CONCLUSIONS

The difficulty of understanding ·color fundus images lies on the fact that the image qualities are not so good as those of other medical images and the variety of colors and blood vessel shapes is very large. There is still a long distance to get the full automatic understanding of color ocular fundus images. We did not discussed the automatic diagnosis of arterio-venous crossing phenomena. In that process we will have to implement semantic level knowledge on symptoms of them into diag­ nostic algorithms.

ACKNOWLEDGEMENTS

The authors wish to thank Dr.s A.Tojo and Y.Shirai of the Electro Technical Laboratory for their helpful advice. Thanks are also due to Dr. S.Nakahara and Mr. S.Ikebata for their hospitality and encouragement.