{"type":"ScholarlyArticle","title":"Pay Attention to Devils: A Photometric Stereo Network for Better Details","authors":[{"name":"Yakun Ju","slug":"yakun-ju","given":"Yakun","family":"Ju","affiliation":"Department of Computer Science and Technology , Ocean University of China"},{"name":"Kin-Man Lam","slug":"kin-man-lam","given":"Kin-Man","family":"Lam","affiliation":"Department of Electronic and Information Engineering , The Hong Kong Polytechnic University"},{"name":"Yang Chen","slug":"yang-chen","given":"Yang","family":"Chen","affiliation":"Department of Computer Science and Technology , Ocean University of China"},{"name":"Lin Qi","slug":"lin-qi","given":"Lin","family":"Qi","affiliation":"Department of Computer Science and Technology , Ocean University of China"},{"name":"Junyu Dong","slug":"junyu-dong","given":"Junyu","family":"Dong","affiliation":"Department of Computer Science and Technology , Ocean University of China"}],"abstract":"We present an attention-weighted loss in a photometric stereo neural network to improve 3D surface recovery accuracy in complex-structured areas, such as edges and crinkles, where existing learning-based methods often failed. Instead of using a uniform penalty for all pixels, our method employs the attention-weighted loss learned in a self-supervise manner for each pixel, avoiding blurry reconstruction result in such difficult regions. The network first estimates a surface normal map and an adaptive attention map, and then the latter is used to calculate a pixel-wise attention-weighted loss that focuses on complex regions. In these regions, the attention-weighted loss applies higher weights of the detail-preserving gradient loss to produce clear surface reconstructions. Experiments on real datasets show that our approach significantly outperforms traditional photometric stereo algorithms and state-of-the-art learning-based methods.","doi":"10.24963/ijcai.2020/97","url":"https://ijcai.org/proceedings/2020/97","pdf":"https://ijcai.org/proceedings/2020/0097.pdf","markdown":"https://ijcai.org/proceedings/2020/97.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/97","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":97,"pages":{"first":694,"last":700},"keywords":["Computer Vision: 2D and 3D Computer Vision","Computer Vision: Computational Photography, Photometry, Shape from X","Machine Learning Applications: Applications of Supervised Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24239/file_24239.m3u8","length_sec":294,"provider":"hls"}],"references":[{"position":1,"raw":"[Alldrin et al., 2008] Neil Alldrin, Todd Zickler, and David Kriegman. Photometric stereo with non-parametric and spatially-varying reflectance. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8. IEEE, 2008.","title":"Photometric stereo with non-parametric and spatially-varying reflectance","authors":"Neil Alldrin, Todd Zickler, David Kriegman","year":"2008","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Guanying Chen, Kai Han, and Kwan- Yee K Wong. Ps-fcn: A flexible learning framework for photometric stereo. In Proceedings of the European Con- ference on Computer Vision (ECCV), pages 3-18, 2018.","title":"Ps-fcn: A flexible learning framework for photometric stereo","authors":"Guanying Chen, Kai Han, Kwan-Yee K Wong","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2019] Guanying Chen, Kai Han, Boxin Shi, Yasuyuki Matsushita, and Kwan-Yee K Wong. Self- calibrating deep photometric stereo networks. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 8739-8747, 2019.","title":"Selfcalibrating deep photometric stereo networks","authors":"Guanying Chen, Kai Han, Boxin Shi, Yasuyuki Matsushita, Kwan-Yee K Wong","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chung and Jia, 2008] Hin-Shun Chung and Jiaya Jia. Ef- ficient photometric stereo on glossy surfaces with wide specular lobes. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8. IEEE, 2008.","title":"Efficient photometric stereo on glossy surfaces with wide specular lobes","authors":"Chung, Jia ; Hin-Shun Chung, Jiaya Jia","year":"2008","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Einarsson et al., 2006] Per Einarsson, Charles-Felix Chabert, Andrew Jones, Wan-Chun Ma, Bruce Lamond, Tim Hawkins, Mark Bolas, Sebastian Sylwan, and Paul Debevec. Relighting human locomotion with flowed re- flectance fields. In Proceedings of the 17th Eurographics conference on Rendering Techniques, pages 183-194. Eurographics Association, 2006.","title":"Relighting human locomotion with flowed reflectance fields","authors":"Einarsson","year":"2006","venue":"Proceedings of the 17th Eurographics conference on Rendering Techniques","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Holroyd et al., 2008] Michael Holroyd, Jason Lawrence, Greg Humphreys, and Todd Zickler. A photometric ap- proach for estimating normals and tangents. In ACM Transactions on Graphics (TOG), volume 27, page 133. ACM, 2008.","title":"A photometric approach for estimating normals and tangents","authors":"Michael Holroyd, Jason Lawrence, Greg Humphreys, Todd Zickler","year":"2008","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ikehata and Aizawa, 2014] Satoshi Ikehata and Kiyoharu Aizawa. Photometric stereo using constrained bivariate regression for general isotropic surfaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2179-2186, 2014.","title":"Photometric stereo using constrained bivariate regression for general isotropic surfaces","authors":"Aizawa Ikehata, Satoshi Ikehata, Kiyoharu Aizawa","year":"2014","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ikehata et al., 2012] Satoshi Ikehata, David Wipf, Yasuyuki Matsushita, and Kiyoharu Aizawa. Robust photometric stereo using sparse regression. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 318- 325. IEEE, 2012.","title":"Robust photometric stereo using sparse regression","authors":"Satoshi Ikehata, David Wipf, Yasuyuki Matsushita, Kiyoharu Aizawa","year":"2012","venue":"2012 IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jian et al., 2019] Muwei Jian, Junyu Dong, Maoguo Gong, Hui Yu, Liqiang Nie, Yilong Yin, and Kin-Man Lam. Learning the traditional art of chinese calligraphy via three-dimensional reconstruction and assessment. IEEE Transactions on Multimedia, 2019.","title":"Learning the traditional art of chinese calligraphy via three-dimensional reconstruction and assessment","authors":"Jian","year":"2019","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Johnson and Adelson, 2011] Micah K Johnson and Ed- ward H Adelson. Shape estimation in natural illumination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2553-2560, 2011.","title":"Shape estimation in natural illumination","authors":"Johnson, Adelson, Micah K Johnson, Edward H Adelson","year":"2011","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ju et al., 2019] Yakun Ju, Xinghui Dong, Yingyu Wang, Lin Qi, and Junyu Dong. A dual-cue network for multispec- tral photometric stereo. Pattern Recognition, page 107162, 2019.","title":"A dual-cue network for multispectral photometric stereo","authors":"Ju","year":"2019","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Matusik et al., 2003] Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. A data-driven re- flectance model. ACM Transactions on Graphics, 2003.","title":"A data-driven reflectance model","authors":"Wojciech Matusik, Hanspeter Pfister, Matt Brand, Leonard Mcmillan","year":"2003","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ruiters and Klein, 2009] Roland Ruiters and Reinhard Klein. Heightfield and spatially varying brdf reconstruc- tion for materials with interreflections. In Computer Graphics Forum, volume 28, pages 513-522. Wiley Online Library, 2009.","title":"Heightfield and spatially varying brdf reconstruction for materials with interreflections","authors":"Klein Ruiters, Roland Ruiters, Reinhard Klein","year":"2009","venue":"Computer Graphics Forum","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Santo et al., 2017] Hiroaki Santo, Masaki Samejima, Yusuke Sugano, Boxin Shi, and Yasuyuki Matsushita. Deep photometric stereo network. In Proceedings of the IEEE International Conference on Computer Vision, pages 501-509, 2017.","title":"Deep photometric stereo network","authors":"Hiroaki Santo, Masaki Samejima, Yusuke Sugano, Boxin Shi, Yasuyuki Matsushita","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shi et al., 2014] Boxin Shi, Ping Tan, Yasuyuki Matsushita, and Katsushi Ikeuchi. Bi-polynomial modeling of low- frequency reflectances. IEEE transactions on pat- tern analysis and machine intelligence, 36(6):1078-1091, 2014.","title":"Bi-polynomial modeling of lowfrequency reflectances","authors":"Boxin Shi, Ping Tan, Yasuyuki Matsushita, Katsushi Ikeuchi","year":"2014","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shi et al., 2019] B Shi, Z Mo, Z Wu, D Duan, SK Yeung, and P Tan. A benchmark dataset and evaluation for non- lambertian and uncalibrated photometric stereo. IEEE transactions on pattern analysis and machine intelligence, 41(2):271-284, 2019.","title":"A benchmark dataset and evaluation for nonlambertian and uncalibrated photometric stereo","authors":"Z Shi, Z Mo, D Wu, Duan, Yeung, Tan","year":"2019","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Taniai and Maehara, 2018] Tatsunori Taniai and Takanori Maehara. Neural inverse rendering for general reflectance photometric stereo. In International Conference on Ma- chine Learning, pages 4864-4873, 2018.","title":"Neural inverse rendering for general reflectance photometric stereo","authors":"Maehara Taniai, Tatsunori Taniai, Takanori Maehara","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ummenhofer et al., 2017] Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, and Thomas Brox. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5038-5047, 2017.","title":"Demon: Depth and motion network for learning monocular stereo","authors":"Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, Thomas Brox","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Woodham, 1980] R. J Woodham. Photometric method for determining surface orientation from multiple images. Op- tical Engineering, 19(1):139-144, 1980.","title":"Photometric method for determining surface orientation from multiple images","authors":"R J Woodham","year":"1980","venue":"Optical Engineering","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu and Tang, 2009] Tai-Pang Wu and Chi-Keung Tang. Photometric stereo via expectation maximization. IEEE transactions on pattern analysis and machine intelligence, 32(3):546-560, 2009.","title":"Photometric stereo via expectation maximization","authors":"Tang ; ] Tai-Pang Wu, Chi-Keung Wu, Tang","year":"2009","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2010] Lun Wu, Arvind Ganesh, Boxin Shi, Ya- suyuki Matsushita, Yongtian Wang, and Yi Ma. Robust photometric stereo via low-rank matrix completion and re- covery. In Asian Conference on Computer Vision, pages 703-717. Springer, 2010.","title":"Robust photometric stereo via low-rank matrix completion and recovery","authors":"Lun Wu, Arvind Ganesh, Boxin Shi, Yasuyuki Matsushita, Yongtian Wang, Yi Ma","year":"2010","venue":"Asian Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}