{"type":"ScholarlyArticle","title":"Structure-Aware Residual Pyramid Network for Monocular Depth Estimation","authors":[{"name":"Xiaotian Chen","slug":"xiaotian-chen","given":"Xiaotian","family":"Chen","affiliation":"National Engineering Laboratory for Brain-inspired Intelligence Technology and Application University of Science and Technology of China"},{"name":"Xuejin Chen","slug":"xuejin-chen","given":"Xuejin","family":"Chen","affiliation":"National Engineering Laboratory for Brain-inspired Intelligence Technology and Application University of Science and Technology of China"},{"name":"Zheng-Jun Zha","slug":"zheng-jun-zha","given":"Zheng-Jun","family":"Zha","affiliation":"National Engineering Laboratory for Brain-inspired Intelligence Technology and Application University of Science and Technology of China"}],"abstract":"Monocular depth estimation is an essential task for scene understanding. The underlying structure of objects and stuff in a complex scene is critical to recovering accurate and visually-pleasing depth maps. Global structure conveys scene layouts, while local structure reflects shape details. Recently developed approaches based on convolutional neural networks (CNNs) significantly improve the performance of depth estimation. However, few of them take into account multi-scale structures in complex scenes. In this paper, we propose a Structure-Aware Residual Pyramid Network (SARPN) to exploit multi-scale structures for accurate depth prediction. We propose a Residual Pyramid Decoder (RPD) which expresses global scene structure in upper levels to represent layouts, and local structure in lower levels to present shape details. At each level, we propose Residual Refinement Modules (RRM) that predict residual maps to progressively add finer structures on the coarser structure predicted at the upper level. In order to fully exploit multi-scale image features, an Adaptive Dense Feature Fusion (ADFF) module, which adaptively fuses effective features from all scales for inferring structures of each scale, is introduced. Experiment results on the challenging NYU-Depth v2 dataset demonstrate that our proposed approach achieves state-of-the-art performance in both qualitative and quantitative evaluation. The code is available at https://github.com/Xt-Chen/SARPN.","doi":"10.24963/ijcai.2019/98","url":"https://ijcai.org/proceedings/2019/98","pdf":"https://ijcai.org/proceedings/2019/0098.pdf","markdown":"https://ijcai.org/proceedings/2019/98.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/98","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":98,"pages":{"first":694,"last":700},"keywords":["Computer Vision: 2D and 3D Computer Vision","Computer Vision: Computational Photography, Photometry, Shape from X"],"videos":[],"references":[{"position":1,"raw":"[Dai et al., 2017] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5828-5839, 2017.","title":"ScanNet: Richly-annotated 3D reconstructions of indoor scenes","authors":"Angela Dai, X Angel, Manolis Chang, Maciej Savva, Thomas Halber, Matthias Funkhouser, Nießner","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Eigen and Fergus, 2015] David Eigen and Rob Fergus. Pre- dicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In IEEE International Conference on Computer Vision, pages 2650-2658, 2015.","title":"Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture","authors":"Fergus ; Eigen, David Eigen, Rob Fergus","year":"2015","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Eigen et al., 2014] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Advances in Neural Information Processing Systems, pages 2366-2374, 2014.","title":"Depth map prediction from a single image using a multi-scale deep network","authors":"David Eigen, Christian Puhrsch, Rob Fergus","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2018] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In IEEE Conference on Computer Vision and Pattern Recog- nition, pages 2002-2011, 2018.","title":"Deep ordinal regression network for monocular depth estimation","authors":"Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, Dacheng Tao","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hao et al., 2018] Zhixiang Hao, Yu Li, Shaodi You, and Feng Lu. Detail preserving depth estimation from a sin- gle image using attention guided networks. In 3DV, pages 304-313. IEEE, 2018.","title":"Detail preserving depth estimation from a single image using attention guided networks","authors":"Zhixiang Hao, Yu Li, Shaodi You, Feng Lu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In IEEE Conference on Com- puter Vision and Pattern Recognition, pages 7132-7141, 2018.","title":"Squeezeand-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2019] Junjie Hu, Mete Ozay, Yan Zhang, and Takayuki Okatani. Revisiting single image depth estima- tion: Toward higher resolution maps with accurate object boundaries. In IEEE Winter Conference on Applications of Computer Vision, 2019.","title":"Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries","authors":"Junjie Hu, Mete Ozay, Yan Zhang, Takayuki Okatani","year":"2019","venue":"IEEE Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiao et al., 2018] Jianbo Jiao, Ying Cao, Yibing Song, and Rynson Lau. Look deeper into depth: Monocular depth estimation with semantic booster and attention-driven loss. In European Conference on Computer Vision, pages 53- 69, 2018.","title":"Look deeper into depth: Monocular depth estimation with semantic booster and attention-driven loss","authors":"Jianbo Jiao, Ying Cao, Yibing Song, Rynson Lau","year":"2018","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ladicky et al., 2014] Lubor Ladicky, Jianbo Shi, and Marc Pollefeys. Pulling things out of perspective. In IEEE Conference on Computer Vision and Pattern Recognition, pages 89-96, 2014.","title":"Pulling things out of perspective","authors":"Lubor Ladicky, Jianbo Shi, Marc Pollefeys","year":"2014","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Laina et al., 2016] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual net- works. In 3DV, pages 239-248. IEEE, 2016.","title":"Deeper depth prediction with fully convolutional residual networks","authors":"Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, Nassir Navab","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2015] Bo Li, Chunhua Shen, Yuchao Dai, Anton Van Den Hengel, and Mingyi He. Depth and surface nor- mal estimation from monocular images using regression on deep features and hierarchical crfs. In IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 1119-1127, 2015.","title":"Depth and surface normal estimation from monocular images using regression on deep features and hierarchical crfs","authors":"Bo Li, Chunhua Shen, Yuchao Dai, Anton Van Den, Mingyi Hengel, He","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3431-3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Long","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Park et al., 2017] Seong-Jin Park, Ki-Sang Hong, and Se- ungyong Lee. RDFNet: RGB-D multi-level residual fea- ture fusion for indoor semantic segmentation. In IEEE In- ternational Conference on Computer Vision, pages 4980- 4989, 2017.","title":"RDFNet: RGB-D multi-level residual feature fusion for indoor semantic segmentation","authors":"Seong-Jin Park, Ki-Sang Hong, Seungyong Lee","year":"2017","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De- Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.","title":"Automatic differentiation in pytorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De-Vito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Qi et al., 2018] Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia. Geonet: Geometric neural network for joint depth and surface normal estimation. In IEEE Conference on Computer Vision and Pattern Recog- nition, pages 283-291, 2018.","title":"Geonet: Geometric neural network for joint depth and surface normal estimation","authors":"Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, Jiaya Jia","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qian et al., 2014] Chen Qian, Xiao Sun, Yichen Wei, Xi- aoou Tang, and Jian Sun. Realtime and robust hand track- ing from depth. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1106-1113, 2014.","title":"Realtime and robust hand tracking from depth","authors":"Chen Qian, Xiao Sun, Yichen Wei, Xiaoou Tang, Jian Sun","year":"2014","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Silberman et al., 2012] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgb-d images. In European Conference on Computer Vision, pages 746-760, 2012.","title":"Indoor segmentation and support inference from rgb-d images","authors":"Nathan Silberman, Derek Hoiem, Pushmeet Kohli, Rob Fergus","year":"2012","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Song et al., 2015] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 567- 576, 2015.","title":"Sun rgb-d: A rgb-d scene understanding benchmark suite","authors":"Song","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2017] Dan Xu, Elisa Ricci, Wanli Ouyang, Xiao- gang Wang, and Nicu Sebe. Multi-scale continuous CRFs as sequential deep networks for monocular depth estima- tion. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5354-5362, 2017.","title":"Multi-scale continuous CRFs as sequential deep networks for monocular depth estimation","authors":"Dan Xu, Elisa Ricci, Wanli Ouyang, Xiaogang Wang, Nicu Sebe","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xu et al., 2018a] Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. Pad-net: Multi-tasks guided prediction- and-distillation network for simultaneous depth estimation and scene parsing. In IEEE Conference on Computer Vi- sion and Pattern Recognition, 2018.","title":"Pad-net: Multi-tasks guided predictionand-distillation network for simultaneous depth estimation and scene parsing","authors":"Xu","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2018b] Dan Xu, Wei Wang, Hao Tang, Hong Liu, Nicu Sebe, and Elisa Ricci. Structured attention guided convolutional neural fields for monocular depth estima- tion. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3917-3925, 2018.","title":"Structured attention guided convolutional neural fields for monocular depth estimation","authors":"Xu","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2018] Zhenyu Zhang, Zhen Cui, Chunyan Xu, Zequn Jie, Xiang Li, and Jian Yang. Joint task-recursive learning for semantic segmentation and depth estimation. In European Conference on Computer Vision, 2018.","title":"Joint task-recursive learning for semantic segmentation and depth estimation","authors":"Zhenyu Zhang, Zhen Cui, Chunyan Xu, Zequn Jie, Xiang Li, Jian Yang","year":"2018","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zheng et al., 2018] Kecheng Zheng, Zheng-Jun Zha, Yang Cao, Xuejin Chen, and Feng Wu. LA-Net: Layout-aware dense network for monocular depth estimation. In ACM Multimedia Conference on Multimedia Conference, pages 1381-1388, 2018.","title":"LA-Net: Layout-aware dense network for monocular depth estimation","authors":"Kecheng Zheng, Zheng-Jun Zha, Yang Cao, Xuejin Chen, Feng Wu","year":"2018","venue":"ACM Multimedia Conference on Multimedia Conference","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}