{"type":"ScholarlyArticle","title":"DeepFacade: A Deep Learning Approach to Facade Parsing","authors":[{"name":"Hantang Liu","slug":"hantang-liu","given":"Hantang","family":"Liu","affiliation":"College of Computer Science , Zhejiang University , China"},{"name":"Jialiang Zhang","slug":"jialiang-zhang","given":"Jialiang","family":"Zhang","affiliation":"College of Computer Science , Zhejiang University , China"},{"name":"Jianke Zhu","slug":"jianke-zhu","given":"Jianke","family":"Zhu","affiliation":"College of Computer Science , Zhejiang University , China; Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies"},{"name":"Steven C. H. Hoi","slug":"steven-c-h-hoi","given":"Steven C. H.","family":"Hoi","affiliation":"School of Information Systems , Singapore Management University , Singapore"}],"abstract":"The parsing of building facades is a key component to the problem of 3D street\r\n  scenes reconstruction, which is long desired in computer vision. In this\r\n  paper, we propose a deep learning based method for segmenting a facade into\r\n  semantic categories. Man-made structures often present the characteristic of\r\n  symmetry. Based on this observation, we propose a symmetric regularizer for\r\n  training the neural network. Our proposed method can make use of both the\r\n  power of deep neural networks and the structure of man-made architectures. We\r\n  also propose a method to refine the segmentation results using bounding boxes\r\n  generated by the Region Proposal Network. We test our method by training a\r\n  FCN-8s network with the novel loss function. Experimental results show that\r\n  our method has outperformed previous state-of-the-art methods significantly on\r\n  both the ECP dataset and the eTRIMS dataset. As far as we know, we are the\r\n  first to employ end-to-end deep convolutional neural network on full image\r\n  scale in the task of building facades parsing.","doi":"10.24963/ijcai.2017/320","url":"https://ijcai.org/proceedings/2017/320","pdf":"https://ijcai.org/proceedings/2017/0320.pdf","markdown":"https://ijcai.org/proceedings/2017/320.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/320","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":320,"pages":{"first":2301,"last":2307},"keywords":["Machine Learning: Deep Learning","Robotics and Vision: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2016] Liang-Chieh Chen, George Papandreou, Ia- sonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Se- mantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv:1606.00915, 2016.","title":"Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cohen et al., 2014] Andrea Cohen, Alexander G. Schwing, and Marc Pollefeys. Efficient structured parsing of facades using dy- namic programming. June 2014.","title":"Efficient structured parsing of facades using dynamic programming","authors":"Andrea Cohen, Alexander G Schwing, Marc Pollefeys","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. IEEE, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong et al., 2014] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision, pages 184-199. Springer, 2014.","title":"Learning a deep convolutional network for image super-resolution","authors":"Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gatys et al., 2015] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576, 2015.","title":"A neural algorithm of artistic style","authors":"Leon A Gatys, Alexander S Ecker, Matthias Bethge","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Han and Zhu, 2009] Feng Han and Song-Chun Zhu. Bottom- up/top-down image parsing with attribute grammar. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 31(1):59- 73, 2009.","title":"Bottomup/top-down image parsing with attribute grammar","authors":"Zhu ; ] Feng Han, Song-Chun Han, Zhu","year":"2009","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 1026- 1034, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by re- ducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.","title":"Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Korč and Förstner, 2009] F. Korč and W. Förstner. eTRIMS Image Database for interpreting images of man-made scenes. Technical Report TR-IGG-P-2009-01, April 2009.","title":"eTRIMS Image Database for interpreting images of man-made scenes","authors":"F Korč, W Förstner","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Ge- offrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3431-3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mathias et al., 2011] Markus Mathias, Andelo Martinovic, Julien Weissenberg, Simon Haegler, and Luc Van Gool. Automatic ar- chitectural style recognition. ISPRS-International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sci- ences, 3816:171-176, 2011.","title":"Automatic architectural style recognition","authors":"Markus Mathias, Andelo Martinovic, Julien Weissenberg, Simon Haegler, Luc Van Gool","year":"2011","venue":"ISPRS-International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mathias et al., 2016] Markus Mathias, Andelo Martinovic, and Luc Van Gool. ATLAS: A three-layered approach to facade pars- ing. International Journal of Computer Vision, 118(1):22-48, 2016.","title":"ATLAS: A three-layered approach to facade parsing","authors":"Markus Mathias, Andelo Martinovic, Luc Van Gool","year":"2016","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Müller et al., 2007] Pascal Müller, Gang Zeng, Peter Wonka, and Luc Van Gool. Image-based procedural modeling of facades. ACM Transactions on Graphics (TOG), 26(3):85, 2007.","title":"Image-based procedural modeling of facades","authors":"Pascal Müller, Gang Zeng, Peter Wonka, Luc Van Gool","year":"2007","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Recky et al., 2011] Michal Recky, Andreas Wendel, and Franz Leberl. Façade segmentation in a multi-view scenario. In 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission, pages 358-365. IEEE, 2011.","title":"Façade segmentation in a multi-view scenario","authors":"Recky","year":"2011","venue":"2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ripperda and Brenner, 2006] Nora Ripperda and Claus Brenner. Reconstruction of façade structures using a formal grammar and rjmcmc. In Joint Pattern Recognition Symposium, pages 750- 759. Springer, 2006.","title":"Nora Ripperda and Claus Brenner. Reconstruction of façade structures using a formal grammar and rjmcmc","authors":"Brenner Ripperda","year":"2006","venue":"Joint Pattern Recognition Symposium","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schmitz and Mayer, 2016] Matthias Schmitz and Helmut Mayer. A convolutional network for semantic facade segmentation and interpretation. ISPRS-International Archives of the Photogram- metry, Remote Sensing and Spatial Information Sciences, pages 709-715, 2016.","title":"A convolutional network for semantic facade segmentation and interpretation","authors":"Mayer ; Schmitz, Matthias Schmitz, Helmut Mayer","year":"2016","venue":"ISPRS-International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale im- age recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Teboul et al., 2010] Olivier Teboul, Loic Simon, Panagiotis Kout- sourakis, and Nikos Paragios. Segmentation of building facades using procedural shape priors. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 3105- 3112. IEEE, 2010.","title":"Segmentation of building facades using procedural shape priors","authors":"Olivier Teboul, Loic Simon, Panagiotis Koutsourakis, Nikos Paragios","year":"2010","venue":"Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Teboul et al., 2011] Olivier Teboul, Iasonas Kokkinos, Loic Si- mon, Panagiotis Koutsourakis, and Nikos Paragios. Shape gram- mar parsing via reinforcement learning. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 2273-2280. IEEE, 2011.","title":"Shape grammar parsing via reinforcement learning","authors":"Olivier Teboul, Iasonas Kokkinos, Loic Simon, Panagiotis Koutsourakis, Nikos Paragios","year":"2011","venue":"Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wendel et al., 2010] Andreas Wendel, Michael Donoser, and Horst Bischof. Unsupervised facade segmentation using repet- itive patterns. In Joint Pattern Recognition Symposium, pages 51-60. Springer, 2010.","title":"Unsupervised facade segmentation using repetitive patterns","authors":"Andreas Wendel, Michael Donoser, Horst Bischof","year":"2010","venue":"Joint Pattern Recognition Symposium","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xie et al., 2012] Junyuan Xie, Linli Xu, and Enhong Chen. Image denoising and inpainting with deep neural networks. In Advances in Neural Information Processing Systems, pages 341-349, 2012.","title":"Image denoising and inpainting with deep neural networks","authors":"Junyuan Xie, Linli Xu, Enhong Chen","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang and Förstner, 2011] Michael Ying Yang and Wolfgang Först- ner. Regionwise classification of building facade images. In Pho- togrammetric image analysis, pages 209-220. Springer, 2011.","title":"Regionwise classification of building facade images","authors":"Michael Ying, Yang, Wolfgang Förstner","year":"2011","venue":"Photogrammetric image analysis","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2010] Peng Zhao, Tian Fang, Jianxiong Xiao, Honghui Zhang, Qinping Zhao, and Long Quan. Rectilinear pars- ing of architecture in urban environment. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 342-349. IEEE, 2010.","title":"Rectilinear parsing of architecture in urban environment","authors":"Peng Zhao, Tian Fang, Jianxiong Xiao, Honghui Zhang, Qinping Zhao, Long Quan","year":"2010","venue":"Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}