{"type":"ScholarlyArticle","title":"Non-Lambertian Multispectral Photometric Stereo via Spectral Reflectance Decomposition","authors":[{"name":"Jipeng Lv","slug":"jipeng-lv","given":"Jipeng","family":"Lv","affiliation":"Peking University"},{"name":"Heng Guo","slug":"heng-guo","given":"Heng","family":"Guo","affiliation":"Beijing University of Posts and Telecommunications; Osaka University"},{"name":"Guanying Chen","slug":"guanying-chen","given":"Guanying","family":"Chen","affiliation":"The Chinese University of Hong Kong , Shenzhen"},{"name":"Jinxiu Liang","slug":"jinxiu-liang","given":"Jinxiu","family":"Liang","affiliation":"Peking University"},{"name":"Boxin Shi","slug":"boxin-shi","given":"Boxin","family":"Shi","affiliation":"Peking University"}],"abstract":"Multispectral photometric stereo (MPS) aims at recovering the surface normal of a scene from a single-shot multispectral image captured under multispectral illuminations. Existing MPS methods adopt the Lambertian reflectance model to make the problem tractable, but it greatly limits their application to real-world surfaces. In this paper, we propose a deep neural network named NeuralMPS to solve the MPS problem under non-Lambertian spectral reflectances. Specifically, we present a spectral reflectance decomposition model to disentangle the spectral reflectance into a geometric component and a spectral component. With this decomposition, we show that the MPS problem for surfaces with a uniform material is equivalent to the conventional photometric stereo (CPS) with unknown light intensities. In this way, NeuralMPS reduces the difficulty of the non-Lambertian MPS problem by leveraging the well-studied non-Lambertian CPS methods. Experiments on both synthetic and real-world scenes demonstrate the effectiveness of our method.","doi":"10.24963/ijcai.2023/139","url":"https://ijcai.org/proceedings/2023/139","pdf":"https://ijcai.org/proceedings/2023/0139.pdf","markdown":"https://ijcai.org/proceedings/2023/139.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/139","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":139,"pages":{"first":1249,"last":1257},"keywords":["Computer Vision: CV: 3D computer vision","Computer Vision: CV: Computational photography"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2011a] Robert Anderson, Björn Stenger, and Roberto Cipolla. Augmenting depth camera output using pho- tometric stereo. Machine Vision and Applications, 2011.","title":"Augmenting depth camera output using photometric stereo","authors":"Anderson","year":"2011","venue":"Machine Vision and Applications","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2011b] Robert Anderson, Björn Stenger, and Roberto Cipolla. Color photometric stereo for multicolored sur- faces. In Proc. of International Conference on Computer Vision (ICCV), pages 2182-2189, 2011.","title":"Color photometric stereo for multicolored surfaces","authors":"Anderson","year":"2011","venue":"Proc. of International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chakrabarti and Sunkavalli, 2016] Ayan Chakrabarti and Kalyan Sunkavalli. Single-image RGB photometric stereo with spatially- varying albedo. In International Conference on 3D Vision (3DV), pages 258-266, 2016.","title":"Single-image RGB photometric stereo with spatiallyvarying albedo","authors":"Ayan Chakrabarti, Kalyan Sunkavalli","year":"2016","venue":"International Conference on 3D Vision (3DV)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018] Guanying Chen, Kai Han, and Kwan-Yee K. Wong. PS-FCN: A flexible learning framework for photomet- ric stereo. In Proc. of European Conference on Computer Vision (ECCV), 2018.","title":"PS-FCN: A flexible learning framework for photometric stereo","authors":"Guanying Chen, Kai Han, K Kwan-Yee, Wong","year":"2018","venue":"Proc. of European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019a] Guanying Chen, Kai Han, Boxin Shi, Ya- suyuki Matsushita, and Kwan-Yee K. Wong. Self-calibrating deep photometric stereo networks. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.","title":"Self-calibrating deep photometric stereo networks","authors":"Chen","year":"2019","venue":"Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2019b] Lixiong Chen, Yinqiang Zheng, Boxin Shi, Art Subpa-Asa, and Imari Sato. A microfacet-based model for photometric stereo with general isotropic reflectance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.","title":"A microfacet-based model for photometric stereo with general isotropic reflectance","authors":"Chen","year":"2019","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dupuy and Jakob, 2018] Jonathan Dupuy and Wenzel Jakob. An adaptive parameterization for efficient material acquisition and rendering. ACM Trans. on Graph., 37(6):1-14, 2018.","title":"An adaptive parameterization for efficient material acquisition and rendering","authors":"Jakob ; Dupuy, Jonathan Dupuy, Wenzel Jakob","year":"2018","venue":"ACM Trans. on Graph","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Enomoto et al., 2020] Kenji Enomoto, Michael Waechter, Kiri- akos N Kutulakos, and Yasuyuki Matsushita. Photometric stereo via discrete hypothesis-and-test search. In Proc. of IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 2311-2319, 2020.","title":"Photometric stereo via discrete hypothesis-and-test search","authors":"Kenji Enomoto, Michael Waechter, Yasuyuki Kiriakos N Kutulakos, Matsushita","year":"2020","venue":"Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Esteban et al., 2011] CH Esteban, G Vogiatzis, and R Cipolla. Overcoming shadows in 3-source photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011.","title":"Overcoming shadows in 3-source photometric stereo","authors":"Esteban","year":"2011","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2021] Heng Guo, Fumio Okura, Boxin Shi, Takuya Fu- natomi, Yasuhiro Mukaigawa, and Yasuyuki Matsushita. Mul- tispectral photometric stereo for spatially-varying spectral re- flectances: A well posed problem? In Proc. of IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 963-971, 2021.","title":"Multispectral photometric stereo for spatially-varying spectral reflectances: A well posed problem?","authors":"Fumio Heng Guo, Boxin Okura, Takuya Shi, Yasuhiro Funatomi, Yasuyuki Mukaigawa, Matsushita","year":"2021","venue":"Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hui and Sankaranarayanan, 2016] Zhuo Hui and Aswin C Sankaranarayanan. Shape and spatially-varying reflectance estimation from virtual exemplars. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(10):2060-2073, 2016.","title":"Shape and spatially-varying reflectance estimation from virtual exemplars","authors":"Zhuo Hui, Aswin C Sankaranarayanan","year":"2016","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ikehata, 2018] Satoshi Ikehata. CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In Proc. of European Conference on Computer Vision (ECCV), 2018.","title":"CNN-PS: CNN-based photometric stereo for general non-convex surfaces","authors":"Satoshi Ikehata, Ikehata","year":"2018","venue":"Proc. of European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johnson and Adelson, 2011] Micah K Johnson and Edward H Adelson. Shape estimation in natural illumination. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.","title":"Shape estimation in natural illumination","authors":"K Micah, Edward H Johnson, Adelson","year":"2011","venue":"Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ju et al., 2018] Yakun Ju, Lin Qi, Huiyu Zhou, Junyu Dong, and Liang Lu. Demultiplexing colored images for multispectral pho- tometric stereo via deep neural networks. IEEE Access, pages 30804-30818, 2018.","title":"Demultiplexing colored images for multispectral photometric stereo via deep neural networks","authors":"Ju","year":"2018","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ju et al., 2020a] Yakun Ju, Xinghui Dong, Yingyu Wang, Lin Qi, and Junyu Dong. A dual-cue network for multispectral photo- metric stereo. Pattern Recognition, page 107162, 2020.","title":"A dual-cue network for multispectral photometric stereo","authors":"Ju","year":"2020","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ju et al., 2020b] Yakun Ju, Lin Qi, Jichao He, Xinghui Dong, Feng Gao, and Junyu Dong. MPS-Net: Learning to recover surface normal for multispectral photometric stereo. Neurocomputing, pages 62-70, 2020.","title":"MPS-Net: Learning to recover surface normal for multispectral photometric stereo","authors":"Ju","year":"2020","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ju et al., 2022] Yakun Ju, Kin-Man Lam, Wuyuan Xie, Huiyu Zhou, Junyu Dong, and Boxin Shi. Deep learning methods for calibrated photometric stereo and beyond: A survey. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.","title":"Deep learning methods for calibrated photometric stereo and beyond: A survey","authors":"Ju","year":"2022","venue":"Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kontsevich et al., 1994] Leonid L Kontsevich, AP Petrov, and IS Vergelskaya. Reconstruction of shape from shading in color images. Journal of the Optical Society of America, pages 1047- 1052, 1994.","title":"Reconstruction of shape from shading in color images","authors":"Leonid L Kontsevich, Petrov, Vergelskaya","year":"1994","venue":"Journal of the Optical Society of America","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ozawa et al., 2018] Keisuke Ozawa, Imari Sato, and Masahiro Ya- maguchi. Single color image photometric stereo for multi- colored surfaces. Computer Vision and Image Understanding, pages 140-149, 2018.","title":"Single color image photometric stereo for multicolored surfaces","authors":"Keisuke Ozawa, Imari Sato, Masahiro Yamaguchi","year":"2018","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rahman et al., 2014] Sejuti Rahman, Antony Lam, Imari Sato, and Antonio Robles-Kelly. Color photometric stereo using a rain- bow light for non-Lambertian multicolored surfaces. In Proc. of Asian Conference on Computer Vision (ACCV), pages 335-350. Springer, 2014.","title":"Color photometric stereo using a rainbow light for non-Lambertian multicolored surfaces","authors":"Rahman","year":"2014","venue":"Proc. of Asian Conference on Computer Vision (ACCV)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Santo et al., 2017] Hiroaki Santo, Masaki Samejima, Yusuke Sug- ano, Boxin Shi, and Yasuyuki Matsushita. Deep photometric stereo network. In Proc. of International Conference on Com- puter Vision Workshops (ICCVW), 2017.","title":"Deep photometric stereo network","authors":"Hiroaki Santo, Masaki Samejima, Yusuke Sugano, Boxin Shi, Yasuyuki Matsushita","year":"2017","venue":"Proc. of International Conference on Computer Vision Workshops (ICCVW)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shafer, 1985] Steven A Shafer. Using color to separate reflection components. Color Research & Application, (4):210-218, 1985.","title":"Using color to separate reflection components","authors":"A Steven, Shafer","year":"1985","venue":"Color Research & Application","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shi et al., 2014] Boxin Shi, Ping Tan, Yasuyuki Matsushita, and Katsushi Ikeuchi. Bi-polynomial modeling of low-frequency re- flectances. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014.","title":"Bi-polynomial modeling of low-frequency reflectances","authors":"Boxin Shi, Ping Tan, Yasuyuki Matsushita, Katsushi Ikeuchi","year":"2014","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shi et al., 2019] Boxin Shi, Zhipeng Mo, Zhe Wu, Dinglong Duan, Sai-Kit Yeung, and Ping Tan. A benchmark dataset and evalu- ation for non-Lambertian and uncalibrated photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2019.","title":"A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo","authors":"Boxin Shi, Zhipeng Mo, Zhe Wu, Dinglong Duan, Sai-Kit Yeung, Ping Tan","year":"2019","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Silver, 1980] William M. Silver. Determining shape and re- flectance using multiple images. PhD thesis, Massachusetts Insti- tute of Technology, 1980.","title":"Determining shape and reflectance using multiple images","authors":"M William, Silver","year":"1980","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Szeliski, 2010] Richard Szeliski. Computer vision: algorithms and applications. Springer Science & Business Media, 2010.","title":"Computer vision: algorithms and applications","authors":"Richard Szeliski","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wiles and Zisserman, 2017] Olivia Wiles and Andrew Zisserman. SilNet: Single-and multi-view reconstruction by learning from silhouettes. In The British Machine Vision Conference, 2017.","title":"SilNet: Single-and multi-view reconstruction by learning from silhouettes","authors":"Zisserman Wiles, Olivia Wiles, Andrew Zisserman","year":"2017","venue":"The British Machine Vision Conference","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Woodham, 1980] Robert J. Woodham. Photometric method for de- termining surface orientation from multiple images. Optical en- gineering, 1980.","title":"Photometric method for determining surface orientation from multiple images","authors":"Robert J Woodham","year":"1980","venue":"Optical engineering","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xie et al., 2014] Wuyuan Xie, Yunbo Zhang, Charlie CL Wang, and Ronald C-K Chung. Surface-from-gradients: An approach based on discrete geometry processing. In Proc. of IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 2195-2202, 2014.","title":"Surface-from-gradients: An approach based on discrete geometry processing","authors":"Wuyuan Xie, Yunbo Zhang, Charlie Cl Wang, C-K Ronald, Chung","year":"2014","venue":"Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yao et al., 2020] Zhuokun Yao, Kun Li, Ying Fu, Haofeng Hu, and Boxin Shi. GPS-NET: Graph-based photometric stereo network. Proc. of Annual Conference on Neural Information Processing Systems (NeurIPS), 33, 2020.","title":"GPS-NET: Graph-based photometric stereo network","authors":"Zhuokun Yao, Kun Li, Ying Fu, Haofeng Hu, Boxin Shi","year":"2020","venue":"Proc. of Annual Conference on Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}