Active Recurrence of Lighting Condition for Fine-Grained Change Detection

1 Introduction

Image-based monitoring and change detection is an important problem in computer vision and machine learning [Feng et al., 2015; Khan et al., 2017]. Since the lighting condition, camera and scene are three key factors jointly determining the appearance of an image, the success of change detection needs to effectively compensate the influence of varied imaging conditions. Unlike classical change detection assuming fixed camera and slowly changed illuminations [Ide et al., 2016], fine-grained change detection aims to find minute changes of high-value scenes, like cultural heritages, in long time intervals under the wild hosting environments [Stent et al., 2016], wherein how to reproduce the reference camera pose and lighting condition is an essential problem and supports many real-world applications, such as accurate surface and material acquisition [Debevec et al., 2000; Ou and Pellacini, 2011], cultural heritage imaging [Tom et al., 2001; Elhabian et al., 2011] and fine-grained change surveillance [Feng et al., 2015; Huang et al., 2017].

∗ is the corresponding author. This work is supported by NSFC 61671325 and 61572354.

Compared to relatively mature camera relocalization (CR) [Williams et al., 2011; Feng et al., 2016; Shi et al., 2018; Miao et al., 2018], lighting recurrence (LR) is more challenging and rarely studied for fine-grained change detection. LR aims to reproduce the lighting condition of a reference image for the same scene with relocalized camera pose. Since slightly varied lightings may cause great false alarmed change detections for the same scene with fixed camera parameters, reproducing the reference lighting condition, either virtually or physically, is a critical issue.

There are some related works that can deal with the LR problem intuitively, including relighting [Ren et al., 2015; 2013; Liu and Do, 2017], reflectance transformation imaging (RTI) [Tom et al., 2001; Huang et al., 2015; Elhabian et al., 2011], and photometric stereo (PS) [Lu et al., 2015; Shi et al., 2016; Midorikawa et al., 2016]. Nevertheless, most of them just passively take multi-illumination images as input to virtually synthesize the target lighting condition, thus belong to synthetic or virtual lighting recurrence (SLR). Besides, to achieve higher reconstruction quality, they usually need hundreds of images with different calibrated illuminations [Ren et al., 2015; Tom et al., 2001], complex lighting models and/or sophisticated optimizations [Ren et al., 2015; Lu et al., 2015], which may inevitably lead to reconstruction errors in the relighted image and cannot work in real-time.

As shown in Fig. 1, in real-world fine-grained change monitoring tasks, we usually get the reference observation by casting a particular near side lighting to highlight the rich microstructures of the object, see Fig. 1(a). Hence, in current observation, slight deviations from the reference lighting condition may cause large differences in the relighted image, see Fig. 1(b)-(d). Specifically, without or inaccurate LR may generate two types of errors in fine-grained change detection, that is, rich or particular 3Dmicrostructures caused false alarms, and different shading caused missing, see Fig. 1(e)- (h) and Fig. 1(i), respectively. 1 Both of them can significantly harm the fine-grained change detection accuracy.

This paper studies ALR that aims to physically reproduce the lighting condition from a single reference image. In contrast to SLR, ALR wants to physically relocalize a real light source to exact the same position of the reference one. Clearly, there are three major differences from ALR to the widely-studied SLR. First, besides the reference image, the input of ALR is just a very limited number of currently captured images under varied light source poses. The number of input images for feasible ALR (usually a dozen) is much less than that of SLR (at least hundreds). Hence, mature but sophisticated SLR methods that need large number of images to guarantee reconstruction accuracy are not applicable. Second, ALR is a dynamic process. The purpose of ALR is to produce reliable online navigation guidance from current lighting pose to the target one, which is not even a problem in any SLR tasks. Third, unlike SLR that prefers accuracy much more than efficiency, instant response and the navigation correctness for each dimension are equally important for the success of ALR. Therefore, ALR is a new problem and no mature SLR methods can be directly used. In fact, to the best of our knowledge, this paper may be the first work on ALR.

1 The rich 3D microstructures of the object under different near side lightings have different shadows, shadings and specular spots that can cause great false alarmed changes. In contrast, some real changes cannot be clearly observed under the lighting whose direction is close to the normal of the surface where the changes occur.

In this paper, we focus on ALR of near point light (NPL) that is a commonly-used realistic lighting device in finegrained change monitoring of cultural heritages. Due to the complexity of NPL model, direct NPL-based ALR is hard to provide online navigation and stable lighting condition estimation. To conquer this problem, as shown in Fig. 2, we propose to use the simple parallel lighting (PL) as an analogy model and compose a navigation ball with two spherical isointensity circles (SICs) indicating the reference and current poses. We theoretically prove that when the two SICs coincide on the navigation ball, the reference shading is strictly recreated on any real object by current light source pose for both PL and realistic NPL models. We also prove the invariance of the proposed ALR strategy to the normal & lighting decomposition ambiguity. As shown in Fig. 1(e)-(i) and our extensive experiments, the proposed ALR works well for both Lambertian and non-Lambertian scenes, and can significantly improve the performance of fine-grained change detection.

2 Active Lighting Recurrence

2.1 ProblemFormulation

As shown in Fig. 2, let Iref ∈ RP be the reference image, P being the pixel number, I = FL(L(ρ | Θ), S) ∈ RP denotes the real lighting/imaging model of the current observation, which is determined by the scene S (e.g., reflectance, normal, specular regions, if any) and the lighting condition function L(ρ | Θ), where Θ and ρ indicate the intrinsic (e.g., radiation power, color temperature, intensity distribution) and extrinsic (i.e., position and orientation) parameters of the light source. Since intrinsic lighting parameters Θ can be easily reproduced using the same light source, we formulate the ALR problem as a dynamic process that progressively reproduces the light source pose ρ,

where mt and λt indicate the t-th ALR navigation direction and magnitude to adjust the light source pose, diag(·) is the diagonalization of a vector. Note, in our formulation, Eq. (1) defines the ALR objective, while Eq. (2) is the progressive ALR strategy, i.e., at t-th iteration, we physically adjust the current light source pose by ∆ρt = diag(λt)mt. Hence, the convergence and goodness of an ALR approach relies on the correctness of diag(λt) andmt, and lim t→∞ diag(λt) = 0.

To provide accurate and instant ALR navigation, we first compose an analogy parallel lighting (apl) based instant navigation ball via Lambertian law and prove its invariance to normal & lighting decomposition ambiguity in Sec. 2.2. We then prove the feasibility of this simple online ALR-apl strategy to the common realistic near point light sources in Sec. 2.3.

2.2 Analogy Parallel Lighting-Based ALR

We use the simple parallel lighting as an analogy model, abbreviated to apl. Let l ∈ R3 denote the parallel lighting vector, whose magnitude and direction indicate the lighting strength and direction, respectively. Let N ∈ RP×3 and R ∈ RP be the normal and grayscale reflectance of the scene, respectively. Under this model, we have L(ρ | Θ) = l and S = {R, N}, with R and N being the scene reflectance and normal, respectively. Thus, the ALR formulation Eqs. (1)-(2) can be reduced to a much simpler ALR-apl problem

where the ◦ is elementwise multiplication.

Thanks to the simplicity of ALR-apl model, it is possible to online calculate both the navigation direction mt and magnitude λt from the reference and current images, Iref and It. Specifically, Fig. 2 shows the working flow of the ALR-apl approach.To get a reliable initialization, we first roughly capture K different side lighting images to form the in-situ captured image set I ic . From I ic and Iref , we obtain scene normal N, reflectance R and reference lighting vector lref . Then, in each ALR-apl iteration, we compose an instant navigation ball B and online calculate the navigation direction mt and magnitude diag(λt) for light source adjustment. We analyze the convergence and efficacy of this process at last.

Initialization

By parallel lighting analogy, we have

where S ∈ RP is shading image that can be obtained by disambiguated intrinsic image decomposition [Zhao et al., 2012], by solving I = R ◦ S. Clearly, from I ic and Iref , we can obtain K +1 shading images (K in-situ captured side lighting images, 1 reference image). They are used to estimate the scene normal N and reference lighting vector lref via a fast state-of-the-art uncalibrated photometric stereo algorithm, LDR [Favaro and Papadhimitri, 2012]. In practice, we only need K ≈ 12 side lighting images for initialization.

Instant Navigation Ball B

Given N and R, current lighting vector lt of It can be easily obtained by solving Eq. (5) in closed-form. To eliminate scene dependency in ALR process, we online render both current and reference lightings, lt and lref , onto a unit sphere Ns , rather than on the real scene normal N. Specifically, let Bref and Bt be the reference and currently rendered images of the unit sphere, i.e., Bref = Ns lref and Bt = Ns lt, where Ns is the sphere normal. From the Lambertian law, we can easily obtain the following proposition about the spherical isointensity sets Cref and Ct, formed by rendered pixels with some particular intensity value, e.g., the median of Bref .

Proposition 1 (SICs & shading equivalence). With analogy parallel lighting and Lambertian law, given an arbitrary lighting condition l, the spherical isointensity set C always forms a circle under the view of lighting direction, which can be named as spherical isointensity circle (SIC). Iff the reference and current SICs Cref and Ct coincide completely, the reference and current images, Iref & It, are the same.

Due to the regular shape of a SIC, we can always make the reference and current SICs, Cref and Ct, coincide on the rendered sphere by adjusting light source pose. Proposition 1 guarantees the shading equivalence between the real reference and current images, once the corresponding SICs coincide. Hence, as shown in Fig. 3(a), we can dynamically compose an instant navigation ball Bt to provide effective instant ALR navigation,

where Bt is the current rendered image on the unit sphere, Oref and Ot are the center coordinates of Cref and Ct, respectively. As shown in Fig. 3(a), In fact, the center coordinate of a SIC indeed indicates the corresponding lighting direction, i.e., Oref = lref

Online Estimation of Navigation Direction mt

Note, see Fig. 3(a), an analogy parallel light source pose ρt can be expressed by vector [rt, θt, φt]T in the light source spherical coordinate system, with rt = ∥∥lref ∥∥ being the lighting strength, θt & φt indicating the lighting direction in the azimuthal and polar axes, respectively. Therefore, according to Proposition 1, we can effectively estimate the ALR navigation direction from current lighting lt toward the target one lref in radial, azimuthal and polar axes as

where Aref = A(Cref ) and At = A(Ct) are the area of SICs Cref and Ct, respectively, A(·) is region area, sgn(·) is the sign function. As illustrated by Fig. 3(a), mt reflects the positive +1 or negative -1 ALR directions along the three axes of light source spherical coordinate system, for the apl model.

Online Estimation of Navigation Magnitude λt

With mt and fast estimation of current lighting vector lt, we can establish a manual control loop with [Aref , θref , φref ]T as desired set-point (SP) and [At, θt, φt]T being the measured process variable (PV). In fact, in our real-world tasks of finegrained change detection of cultural heritages, we mainly relied on this manual ALR to do the job, due to its great portability.

Nevertheless, as shown in Fig. 3(a), we can also effectively estimate the ALR navigation magnitude λt via bisection approaching, which together with mt can enable an automatic ALR process, with the help of a robotic arm. Specifically, given an initial λ0 = [rλ 0 , θλ 0 , φλ0 ]T, where rλ 0 , θλ 0 , φλ0 indicate the initial light source adjustment magnitude in radial, azimuthal and polar axes, respectively. In our experiments, we empirically set rλ 0 = θλ 0 = φλ0 = 3mm in our robotic platform. In t-th (t > 1) ALR iteration, we have

where a ∈ {r, θ, φ} denotes the three independent spherical axes, µ is the speed-up rate of navigation magnitude and is empirically set as 1.2 in our experiments. With Eq. (8), we can efficiently update λt and obtain the ALR increment ∆ρt = diag(λt)mt, which is directly applied on the robotic platform to finish the t-th ALR adjustment.

See Fig. 3(a), Eq. (8) indeed defines a bisection approaching strategy. To understand the convergence of this strategy, we conceptually separate the ALR process into two stages. First, regardless of the spherical position (θt, φt) of the center Ot of current SIC Ct, we first make the area of Ct equal to that of Cref . That is, we first reproduce the target lighting strength of the analogy parallel light source. In this condition, current lighting pose ρt and the reference pose ρref lie on the same great circle of a bigger sphere whose center coincides with that of the scenario coordinate frame. mt indicates the 1D relative position from ρt toward ρref on this great circle.

Hence, the second stage is to use the bisection strategy to approach ρref , by iteratively checking the relative direction of mt and mt-1. That is, on the great circle, if ρt do not reach ρref , we enlarge the step size; if ρt crosses ρref , we bisect the step size. Since in each cases mt guarantees the right direction, it is clear that such strategy can effectively lead ρt toward ρref , i.e., lim t→∞ diag(λt) = 0. See the green curve of

Fig. 3(a) for an example of light source adjustment trajectory.

Algorithm & Implementation Details

Fig. 2 shows the detailed working flow of our ALR-apl approach. This process is terminated by an ALR goodness g that measures the recurrence accuracy by the overlap ratio of Cref and Ct,

If g is good enough, e.g., g > 0.99, the ALR process stops.

Invariance to N & l Decomposition Ambiguity

According to Eq. (5), the N& l decomposition generally subject to an ambiguity matrix Z,

where ˜N ∈ RP×3 and ˜l ∈ R3 are the ground truth normal and lighting condition. Therefore, our initialization also exits an ambiguity matrix Z between the estimated and real scene normal N and lighting direction l, which may influence the correctness of ALR. Fortunately, we prove that if the estimation error of scene normal N is smaller than π3 , the ambiguity matrix Z does not affect the effectiveness and convergence of our ALR approach.

Lemma 1. The ambiguity matrix Z generated by the decomposition of Eq. (5) is a rotation matrix.

Lemma 2. The radial-axis navigation direction, i.e., rm 0 , is independent to Z.

Lemma 3. Let 〈β, e〉 be axis-angle representation of Z. For the azimuthal and polar ALR direction guidance in Eq. (7), if β < π3 , Z does not affect our ALR process. We can faithfully relocalize the light source to reference apl pose.

Brief proof . Lemma 2 ensures the lighting strength can be reproduced, regardless of Z. We need only relocalize Ot to Oref on the unit sphere. In this condition, as shown in Fig. 3(b), light source increment ∆ρt is equivalent to S1: Near-Lambertian S4: Non-Lambertian (specular)

a rotation matrix Tt. Considering Z, we have Tt+1 = TtZP-1 t Z-1 , where Pt is the real light source adjustment realized by robotic arm movement or manual adjustment. Normally, Pt = Tt. So, Tt+1 = TtZT-1 t Z-1 . Then, the conclusion follows by Theorem 1 of [Feng et al., 2016].

The convergence necessary condition β < π3 of Lemma 3 can be easily satisfied by current photometric stereo methods [Favaro and Papadhimitri, 2012], which is empirically verified and discussed in detail in Sec. 3.1.

2.3 ALR under Realistic Near Point Lighting

In fine-grained change detection, near point lighting (NPL) is a commonly-used realistic model that assigns different scene points with distance-related lighting directions,

where Sp is the shading intensity of pixel p, ρ ∈ R3 is the near point light source position, Xp ∈ R3 indicates the spatial coordinate of point p, e is the lighting power. Eq. (11) describes the quadratic attenuation effect of lighting strength w.r.t. the distance between the light source and scene. In the following, we prove that the proposed ALR-apl approach still works for such realistic NPL light source, thus avoiding sophisticated estimation of the NPL parameters. In this case, lref and l in Fig. 2 now denote the equivalent parallel light vector of the near point lighting.

Proposition 2 (SICs & shading equivalence). Under NPL model, if we locate the point light source to an arbitrary position, then the spherical isointensity set C acquired from the rendered ball image B always forms a circle, under the view that light source points to the sphere center. Iff the reference and current SICs Cref and Ct coincide completely, the reference and current images, Iref & It, are the same.

Proposition 3 (ALR navigation equivalence). The ALR approach for the apl model is also applicable to NPL model.

Brief proof . We use superscripts apl and NPL to distinguish the apl model and NPL model. Given any two points p and q, it is easy to verify Bapl p ⋆Bapl q ⇔BNPL p ⋆BNPL q , where ⋆ denotes an operator in {>,<,=}. That is, Bapl p and BNPL p have the same order relation. So, the proposed ALR guidance also applies to near point lighting condition.

3 Experimental Results

We build 7 scenes (S1-7) to evaluate our ALR method and baselines. S1-3 are near-Lambertian scenes. S4-7 are nonLambertian scenes, including many specular (S4, S7), transparent (S5) and cast shadow (S6) regions respectively. For each scene, we collect 20 multi-illumination images by a Canon 5D Mark III camera. We use a small LED bulb as the near point light source to carry out the experiments.

3.1 Convergence and Effectiveness Validation

In our ALR, we use ALR goodness g to determine the best lighting recurrence result. To verify the effectiveness, we record all images and the corresponding g during ALR for S4 and S5, then we compute SSIM and MS-SSIM [Wang et al., 2003] for each image. Fig. 4(a) and (b) show the relation of ALR goodness g and the average (MS-)SSIM for S4 and S5, respectively. Besides, the difference maps (magnified by 10) of reference image and some recorded images during ALR are also shown. Since the minute change of lighting condition may cause large differences in image, it is reasonable that the (MS-)SSIM may decrease slightly sometimes, but the (MS-)SSIM increases with the increase of g in global. It means that g is an effective goodness criterion for our ALR.

We prove that the ambiguity matrix Z does not influence our ALR method if β < π3 , where β is the angle of axis-angle representation of matrix Z. In fact, the condition is easy to be met. To verify this, we introduce the dataset [Xiong et al., 2015] which includes 7 statue scenes (e.g., Cat, Frog, Hippo) and corresponding ground truth normals. Each scene has 20 multi-illumination images. We estimate the scene normal by LDR [Favaro and Papadhimitri, 2012] and calculate the corresponding mean angle error (MAE) for each scene. We find that the average, maximum and minimum MAEs are π 27 , π 15

and π 45 respectively, which are much less than π3 . Therefore, according to Lemma 3, we can confidently say that the ambiguity matrix Z does not influence our ALR method.

Our ALR method needs not large number of images. To verify this, we carry out ALR for 7 scenes (mentioned later) with variable number of multi-illumination images. The 2-th to 5-th rows of Table 1 show the averages of 4 commonlyused criteria MSE, PSNR, SSIM and MS-SSIM under different image numbers. The 6-th row of Table 1 shows the MAE between each estimated scene normal and the one using 100 images. We can see that the MAE reduces with the increase of image number, and the recurrence accuracy of ALR is stable and always well. It also proves that our ALR is effective as long as the estimated scene normal satisfies Lemma 3. In practice, we commonly collect about 20 images for our ALR.

3.2 Quantitative Comparison

To compare with our ALR method, we use PTM [Tom et al., 2001], HSH [Elhabian et al., 2011], LDR [Favaro and Papadhimitri, 2012] as our baselines. PTM [Tom et al., 2001] and HSH [Elhabian et al., 2011] are two image-based relighting methods, and we use a light probe to calibrate the lighting direction for each captured image. LDR [Favaro and Papadhimitri, 2012] is a state-of-the-art uncalibrated photometric stereo method based on parallel lighting model. We carry out the 3 methods using the captured 20 images and generate 20 lighting recurrence images. We use MSE, PSNR, SSIM, MSSSIM [Wang et al., 2003] as accuracy metrics.

Near-Lambertian Scenes

Fig. 5 shows the quantitative comparisons of our ALR method and 3 baselines for near-Lambertian scenes (S1-3). Each node indicates the average evaluation of 20 lighting recurrence results and the up and down bar of each node denotes the variance. Besides, the average of the 4 criteria for all scenes are also shown. Fig. 6 shows the visual comparisons of 4 methods for S1, including the zoom-in regions and corresponding difference maps. We see that our ALR method can generate more accurate lighting recurrence results than baselines for both quantitative and visual comparisons. PTM has the next best recurrence accuracy, and our ALR improves PSNR score by about 29 percent than PTM. Besides, we can see that even if the difference map has been magnified by 10, the difference in image is not obvious.

Non-Lambertian Scenes

As shown in Fig. 5 (S4-7) and Fig. 6 (S4-6), we can also see that our ALR method is more effective than baselines for non-Lambertian scenes. From the zoom-in regions and difference maps in Fig. 5, we find that the baselines can not faithfully reproduce the surface details for non-Lambertian scenes generally, e.g., the missing of specular region for S4 in Fig. 5. Besides, compared with the near-Lambertian scenes, the baselines generally generate mush worse results for the non-Lambertian scenes.

3.3 Real-World Applications

We apply our ALR to actively capture and measure the finegrained changes of ancient murals in two World Cultural Heritage Sites, Dunhuang Mogao Grottoes (Case 1-2) and Weijin Graves (Case 3-6) in Fig. 7. Specifically, given the reference image, we first relocalize current camera pose via ACR [Feng et al., 2016], then we do ALR to physically reproduce the lighting condition of reference image and take current images for evaluation. Besides, we also capture 20 images under different lighting conditions in current observation and carry on PTM [Tom et al., 2001] to generate relighting images for comparison. Fine-grained changes are detected by FGCD algorithm [Feng et al., 2015] for both ALR and PTM results.

Fig. 7 shows 6 real cases of fine-grained changes of ancient murals, which are caused by either natural deteriorations (Case 1-2) or artificial intervention, e.g. mural restoration (Case 3-6), respectively. Refer to the zoom-in regions and corresponding FGCD results, we can clearly see that our ALR can generate much higher F1-Measure and lower FPR errors. This is because that some surface details cannot be faithfully reproduced by PTM, e.g., the cast shadow of Case 2 in Fig. 7. In contrast, our ALR supports much more accurate fine-grained change detection.

4 Conclusion

We have studied a new problem, active lighting recurrence (ALR), that actively reproduces the actual lighting condition of a single reference image. To achieve instant and accurate ALR guidance, we propose a simple yet effective analogy parallel lighting (apl) based ALR approach. We show that the proposed approach works well for the commonly-used realistic near point lighting model, with strict theoretical equivalence and convergence guarantees. Extensive quantitative experiments and challenging real-world tasks on fine-grained change monitoring of cultural heritages verify the effectiveness and superiority of our approach. In the future, we plan to explore full 6D ALR for more general near surface lighting model. We hope this work could encourage further studies of active recurrence of imaging conditions, for both camera and lighting, and their real-world applications. Besides, we are also interested in exploring spherical panorama [Zhao et al., 2015] based active environment lighting recurrence and using region-level strategy [Feng et al., 2010] to further speedup and stabilize the ALR process.