--- title: "Specular Stereo" authors: - "Andrew Blake" doi: "" url: "https://ijcai.org/proceedings/1985/195" pdf: "https://ijcai.org/proceedings/1985/0195.pdf" venue: "Proceedings of the Ninth International Joint Conference on Artificial Intelligence" year: 1985 pages: "973-975" track: "Perception" keywords: --- # Specular Stereo Andrew Blake *Proceedings of the Ninth International Joint Conference on Artificial Intelligence*, pages 973-975. ## Abstract A glossy highlight, viewed stereoscopically, can provide inforrnation about surface shape. For exarnple, highlights appear to lie behind convex surfaces but in front of concave ones . A highlight is a distorted, reflected image of a light source. A ray equation is developed to predict the stereo disparilies generated when a point source of light is reflected in a srnooth, curved surface. This equation can be inverted to infer surface curvature properlies from observed stereo disparilies of the highlighl. To obtain full information about surface curvature in the neighbourhood of the highlight, stereo with two different baselines - or stereo with rnotion parallax - is required. The same ray equation can also be used to predict the monocular appearance of a distributed source. A circular source, for instance, may produce an elliptical specular patch in an image, and the dimensions of the ellipse help to determine surface shape. ## Full text ## 1 INTRODUCTION When the reflectance of a surface has a specular as well as a diffuse component , the viewer may see highlights . Highlights can give extra information about surface shape. Ikeuchi [B] uses photometric stereo with specular surfaces to determine surface orientation. Beck [1] notes that stereo vision might be a ble to perceive highlights on a convex surface as lying beneath the surface. Grimson [7] incorporated lambertian and specular components of reflectance into stereo. But he found that the computation of surface orientation could be numerically unstable. Here a computation is' p roposed ambitious than Grimson's, in that it determine only local surface geometry, points. But it avoids relying on precise about surface reflectance. that is less attempts to a t specular assumptions Instead, the only assumption is that a specularity can be detected in an image, and its position measured. For instance a method like that of Ullman [ 15] could be used. Thereafter specularities are matched in the same way as features in conventional stereo (6,9 ,10] . The disparity of a stereo-matched specular point is then compared with the disparity of any nearby surface features . The basic principle of the surface shape estimation relies on the ·properties of curved mirrors (fig 1 ). To interpret specular stereo, both horizontal and vertical disparilies are used. ldeally, three non­ collinear eyes are needed to obtain full information about local curvature. Alternatively, parallax from a known vertical motion of a viewer, combined with conventional stereo geometry, is just as good. If only a static, stereo view is available then this still yields partial information. This could be combined ## SPECULAR STEREO ## A. Blake Computer Science Deparlrnenl, Edinburgh Universily, Scolland. ![Image](https://ijcai.org/proceedings/1985/fulltext/195/image_000000_2a768db4e3eb90839268946a4d7a1d8147be23b4b3438b85e70521939c896301.png) Figu.re\_ 1: Viewing geo-metry. In a convex -m~rror (a) the image of a distant point source appears behind the mirrar surface . In a concave one (b) the image may appear in front . Study of the ordinary domestic soup spoon should confirm this . with a priori knowledge or measurements from other sources (stereo, shape-from-shading or specular reflection of a distributed source) to fully determine local surface shape. Finally, observe that the path of a light ray from source to viewer can be reversed. Analysis developed to show the effect of moving the viewer also serves for movement of the source . The resulting equa tion is used to predict the appearance in an image of a distributed source under specular reflection in a curved surface . For instance a circular source generally produces an elliptical specularity in the image. The orientation and length of its major and minor axes , in principle, determine local surface shape . ## 2 IMAGING EQUATIONS Equations are given to describe the process of formation of images of specular reflections . Details of derivations are given in (3]. These predict the dependence of observed stereo disparilies on surface and viewing geometry. Certain assumptions about the geometries are made, for the sake of mathematical simplicity. Then the e quations are inverted so that, given viewing geometry and di sparities, local surface geometry can be inferred . ## 2.1 Viewing, surface and reflection geometry The s tereo viewing geometry is shown in fig 2. Figure 2 : Stereo viewing geoTTLetry. Illumination comes from a distant point s ource, in direction L . Rays to left and right eyes lie along vectors V, W, and s trike the surface at points A,B r espectively . Surface normals at A ,B are N,N' respective ly . The vector from A to B i s r , and the baseline lies along vector d . A surface feature is assumed to be present nearby, at C, with position vector F relative to the l eft eye . ![Image](https://ijcai.org/proceedings/1985/fulltext/195/image_000001_d5176badcf45f4c7944e6e14edaeaab111ffcfbb31718e82e948524c47e97b07.png) It is ass umed that the curved surface is locally weil approximated by terms up to 2nd order in a Taylor series (see e q . (4)) . The vectors V,d ,- W, - r form a closed loop, so tha t A coordinate frame is chosen with origin at A, with N= (0,0,1), and with L.V lying in the x - z plane, so tha t where a is the slant of the tangent plane at A . Its tilt direction lies in the x- z plane . Note that if viewing geometry and light- source direction L are known (and the !alter could be obtained as in [11]) then surface s lant and tilt are known : the surface norma l lies in the plane of V,L and bisects them. It is assumed that some feature at point C on the surface, is available near to the specular points A ,B (fig 2) and tha t stereo is able to establish the position o f C. Its position vector F is used to estima te V , the length of the vector V. Assuming tha t C is not too far away from A, so that C lies, approximately, in the tangentplane at A, (V-F) .N = 0 so tha t Sin ce the choice of coordinate frame ensures that gradients vanish (az j ax=az j ay=O) , the surface . in the n e ighbourhoo d of A, is described by where r =(x,y,z), x =(x,y) and H is the (symmetric) hessian matr ix [4] of the surface . Note that r ,d e t c. are 3-dimens ional vectors but x is a 2dimen s ional vec t or, in the xy- plane . Similarly H is a 2x2 matrix operating on x . The law of reflection at A is that where II denotes "is parallel to" . Similarly for the other eye, at B , where w,= - d,+d.tana , wy= -dy and The approximations used above hold good provided löNI «cosa and lx i«Vcosa. This means that surface slant a must not be close to 90°, and that both vergence angle and (angular) disparity should be small. It can be shown that these conditions will usually be satisfied when the stereo baseline is short, so that ld i«Vcosa. To solve equation (7), we note also that so that , provided surface slant a is not near 90° as above , and provided ld i:S: (l /2)V (baseline length less than half viewing di stance), then det(M)fO . In that case, equation (7) can be inverted to give which, in general, can be expec ted to impose 2 constraints on the 3 variables of H . !f so mething is known already about H - say, that the surface is locally cylindrical at A - then it might be possible to determine H completely . ## 2 .2 Disparity measurement The e quations just derived require the vector x to be determined from disparity measurements, as shown in fig 3. Having obtained from the stereo images the angular disparity ö of the specularity as in fig 3, it can be "back-projected" onto the surface to obtain the length x . The assumption that löNI is small is used again to obtain Figure 3: Dispa.rity Tnea.sureTTLents . The specular point is imaged at angular positions SL.SR i n the left and right images respectively . A nearby surface feature i s imaged at AL,AR and provides a disparity reference point on the surface . From these measured positions in the im.age, ö = (S~rAR)-(SL-AR) i s computed - the d~fference between the angular disparilies of the 2 points . Then x = (x,y) = V(ö,seca,öy) . ![Image](https://ijcai.org/proceedings/1985/fulltext/195/image_000002_2d46e54d579e3b64b5201df4b518ee228ace52c41c9fcebb3b9dff3144159c07.png) ## 2.3 Focusing effects Equation (7) predicts that imaging of a specularity can become degenerate (fig 4) . The equation can be rewritten as and lhe condilion for degeneracy is thal del(JIH-1) = 0. The focusing effecl may produce eilher a line or a blob in lhe image: 1. If lhe rank of (JIH-1) is 1 lhe specularily will appear in lhe image as a line. Or else, lhere may be no solulion for x in (7), and nothing of the specularily will be visible. Stevens L13l observes lhal, with infinilely dislanl source and viewer' any line specularily musl lie on a rlane curve in lhe surface - a special case o lhe rank 1 focusing effecl. 2. If lhe rank of JIH-1 is zero, lhal is MH-1=0, then eilher lhe specular refleclion is invisible as above, or il is focussed in 2 dimensions onlo the imaging aperlure, and appears in lhe image as a lar.ge brighl blob. 3. Il can be shown lhal if lhe surface palch is convex, lhe effecl cannol occur . This corresponds lo physical inluilion. Only concave mirrors focus dislanl light sources. ## 2 .4 A source at a finite distance If the source is al a finite lhan infinite as assumed so equation (7) becomes: dislance L, rather far, the imaging The conslanl p=(d,seca+V)/L and clearly, as L-+oo, ![Image](https://ijcai.org/proceedings/1985/fulltext/195/image_000003_ccb850e6e217f9eb71f611857d0d518b0731d57e4ab5576f7d37a3c56d2cee05.png) Figure 4: Degenerate iTTLaging of a specularity. Normal imaging of a specularity (a) becomes degenerate because of the focusing action of the curved surface. The surface may focus onto the viewing aperture ( b) to produce a bright blob or line in the image, or away from the aperture (c) in which case nothing is visible in the image . ## p-+0 which gives lhe infinite source equation (7). Suppose the infinite source compulalion of surface shape is performed, when in facl lhe source is at a finite distance L, how great is the resulting error? The answer is that the error in the curvalure (along a given direction in the xy-plane) is of the order of ± 1/ L . The error is negligible if. assuming a not lo be close to 90°, eilher - lhe light source dislance is !arge compared wilh lhe viewing distance : L»V, or - lhe surface has high curvature : for both principal curvalures, IC; » 1/ L, i=1,2. The firsl case is inluitively reasonable; if the light source is dislant compared wilh the observer dislance V, lhen equalions for an infinite light source can be used with lillle error . Whal is perhaps less obvious is lhe second condilion thal, for highly curved objecls, the source need not be further away lhan the observer . ## 2 .5 Distributed sources The malhemalical model thal has been used so far assumes a poinl source . In praclice the source may be dislribuled, so lhal il subtends some non­ zero solid angle, al lhe surface . Equalion (12), for a source al a finite dislance, is used bul source and viewer positions are inlerchanged. The light ray is reversed . Veclor d now represenls lhe movemenl of source for a fixed viewer posilion . After some rearrangemenl , this yields a new equation, looking rather like (12) but wilh a faclor V / L on lhe righl hand side : It would be most convenient to express lhe shape of lhe image specularily (using angular posilion in lhe image, d) in lerms of source dislribulion (using a new angular variable a) . From (10) x =VPd, and il is straighttorward lo show lhal w =LPa. So now Whal equalion (14) says is lhal lhe viewer sees an image of lhe source thal has undergone a linear transformalion T-1. The effect of lhe lransformation depends on surface shape . For a planar mirrar for example, H=O so that d=-a/(1+p) - an isolropic scaling lhal preserves lhe shape of lhe source . Note thal if lhe source is very dislant, p:::;O and the scaling factor is unity. If lhe angular dimensions of the source are known then, in principle, surface shape may be recovered completely by monocular Observation. For a circular source with slant a=O, the ellipse axes coincide wilh lhe principal curvalure direclions of lhe surface. In general, when a/0 , measuring the lenglh and direclion of ellipse axes ena bles T and hence H to be found from (14) . Note that for a circular source, because of its symmetry, principal curvatures are delermined only up lo sign inversion (approximalely) . ## 3 INFERRING LOCAL SURFACE SHAPE ## 3 .1 Locally cylindrical surface On a surface that is known to be locally cylindrical, equation (7) is sufficient to recover both parameters of local surface shape. For instance, when lhe source is dislributed, a strip shape image­ specularity indicales that the surface may be cylindrical - or at least that one principal curvalure may be much !arger than the other . (This can be deduced from (14) .) The parameters to be delermined are: lhe direclion of the cylinder axis 8 and the radius R . Using (9) : ## 3.2 Spherical surface The knowledge lhal the surface is locally spherical could be derived monocularly, from ( 14), as in the cylindrical case except that rather more musl be known aboul the source - for example, that il is circular . On a spherical surface there is only one parameler lo specify - the radius of curvature R, and from (9): - the second equality being available as a check for consistency of assumptions . ## 3.3 Known orientation of principal axes If the orientation of principal axes about the surface normal, is known then the complete local surface geometry can be obtained. Orientation could be derived monocularly (assuming source shape known) from (14). Rotating coordinates about the z axis, a primed (') frame can be obtained in which H in (9) becomes diagonal: Now, in general, the two diagonal components of H' can be obtained immediately. Experiments with computer generated images have obtained curvature to an accuracy of 10%. ## 3.4 General case In the general case, the surface curvature at the specular point is described by 3 parameters, but the specular stereo measurements yields only 2 constraints. However two additional constraints - 4 in all - are available if a second baselirre is used. The extra baselirre could be derived either from a third sensor, suitably positioned, or from known motion of the viewer (parallax) . Suppose now that there are 2 baselirres d{il,i= 1,2 with corresponding :z:(il,Af(il,w(il,v(i) . Now equation (9), applied once for each baseline, gives and H can be recovered provided X is non-singular . It appears to be impossible to suggest baselirres that guarantee to generate a non-singular X, for all viewing geometries and surfaces . This is because det(X) depends on the surface and the viewing geometry, as well as on the baselines. This is probably best achieved (see [3)) by making the baselirres d(i) fairly near orthogonal, and certainly nowhere near collinear. The disparity measurements give 4 constraints. If H is the only unknown, it is now overdetermined. One could either 1. Test whether the H obtained from (15) is indeed symmetric as a check on validity of assumptions (for example, the validity of the local approximation of (4), over the range of movement of the specular point on the surface) 2. Use a Ieast-squares error method to find the symmetric H that fits the data best. Then H is the solution of linear equations: The error measure IIHX-VII. if it is too !arge, indicates that some assumptions were not valid. ## 4 CONCLUSION Is specular stereo actually usefuJ? We argue that it is. Of course the presence of specularities in the image cannot be guaranteed; specular stereo is not an autonomaus process in the sense that conventional stereo is. Indeed specular stereo itself relies on conventional stereo to provide a disparity reference. In the case of a densely textured surface, conventional stereo with surface fitting [2,5,6,12,14) would be able to give an accurate estimate of surface shape. But for a smooth surface, stereo features may be relatively sparse , and fitting a surface to disparity measurements may be difficult and inaccurate. Then, provided at least one nearby surface feature is available as a disparity reference, specular stereo, tagether with monocular analysis of specularity, provides valuable surface shape information. ## Acknowledgement This work was supported by SERC grant GR/D 1439.6 and by the University of Edinburgh. Thanks are due to G. Brelstaff, A. Zisserman and R. Fisher for valuable discussion. ## References 1. Beck, J . (1972). Surface color perception . Cornell University Press, Ithaca, U.S .. 2. Blake, A. (1984). Reconstructing a visible surface. Proc AAAI conf. 1984, 23-26. 3. Blake, A. (1984). Interring surface shape by specular stereo . Report CSR-179-84, Dept . Computer Science, Edinburgh University. 4. do Carmo , M.P . (1976). Differential geo"Tnetry of curves and surfaces. Prentice Hall, Englewood cliffs, USA . 5. Faugeras, O.D. and Hebert, M. (1983). A 3-D recognition and positioning algorithm using geometrical matehing between primitive surfaces. IJCAI 83, 996-1002. 6. Grimson, W.E .L. ( 1982). Fro"Tn i"Tnages surfaces. MIT Press, Cambridge, USA. to 7. Grimson, W.E.L. (1982) . Binocular shading and visual surface reconstruction . AI Lab . Me"Tno . 697, MIT, Cambridge, USA. 8. Ikeuchi,K. ( 1981 ). Determining surface orientations of specular surfaces by using the photometric stereo method. IEEE trans. PAMI, 3, 6, 661-669 . 9. Marr, D. and Poggio, T. (1979). A computational theory of human stereo viswn. Proc . R . Soc Lond. B, 204, 301-328. 10. Mayhew, J.E.W and Frisby, J .P. (1981). Towards a computational and psychophysical theory of stereopsis. AI Journal, 17, 349-385. 11. Pentland, A.P. (1984). Local shape analysis. IEEE trans . PAMI, March 1984, 170-187 . 12. Potmesil, M. (1983). Generating models of solid objects by matehing 3D surface segments . IJCAI 83, 1089-1093. 13. Stevens,K.A. (1979). Surface perception fro"Tn local analysis of texture and contour. Ph .D. thesis, MIT, USA. 14. Terzopoulos, D. (1983). processes for visual Co7nputer Vision Processing. 24, 52-96 . Multilevel computational surface reconstruction, Craphics and I"Tnage 15. Ullman, S . (1976) . On visual detection of light sources. Biol. Cybernetics, 21, 205-212.