{"type":"ScholarlyArticle","title":"Single-Image 3D Scene Parsing Using Geometric Commonsense","authors":[{"name":"Chengcheng Yu","slug":"chengcheng-yu","given":"Chengcheng","family":"Yu","affiliation":"Department of Statistics , University of California , Los Angeles"},{"name":"Xiaobai Liu","slug":"xiaobai-liu","given":"Xiaobai","family":"Liu","affiliation":"Department of Computer Science , San Diego State Univesity (SDSU) , CA"},{"name":"Song-Chun Zhu","slug":"song-chun-zhu","given":"Song-Chun","family":"Zhu","affiliation":"Department of Statistics , University of California , Los Angeles; (UCLA) , CA"}],"abstract":"This paper presents a unified grammatical framework capable of reconstructing a variety of scene types (e.g., urban, campus, county etc.) from a single input image. The key idea of our approach is to study a novel commonsense reasoning framework that mainly exploits two types of prior knowledges: (i) prior distributions over a single dimension of objects, e.g., that the length of a sedan is about 4.5 meters; (ii) pair-wise relationships between the dimensions of scene entities, e.g., that the length of a sedan is shorter than a bus. These unary or relative geometric knowledge, once extracted, are fairly stable across different types of natural scenes, and are informative for enhancing the understanding of various scenes in both 2D images and 3D world. Methodologically, we propose to construct a hierarchical graph representation as a unified representation of the input image and related geometric knowledge. We formulate these objectives with a unified probabilistic formula and develop a data-driven Monte Carlo method to infer the optimal solution with both bottom-to-up and top-down computations. Results with comparisons on public datasets showed that our method clearly outperforms the alternative methods.","doi":"10.24963/ijcai.2017/649","url":"https://ijcai.org/proceedings/2017/649","pdf":"https://ijcai.org/proceedings/2017/0649.pdf","markdown":"https://ijcai.org/proceedings/2017/649.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/649","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Uncertainty in AI","track":"Main track","number":649,"pages":{"first":4655,"last":4661},"keywords":["Robotics and Vision: Vision and Perception","Uncertainty in AI: Bayesian Networks"],"videos":[],"references":[{"position":1,"raw":"line, 'location', 'co-linear', line, 'location')","title":"location', 'co-linear', line","authors":"Line","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"line, 'direction', 'parallel', line, 'direction')","title":"direction', 'parallel', line","authors":"Line","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"line, 'direction', ''orthogonal, line, 'direction')","title":"orthogonal, line","authors":"Line","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"line, 'location', 'co-planar', planar, 'location')","title":"location', 'co-planar', planar","authors":"Line","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"line, 'direction', 'parallel', planar, 'normal')","title":"direction', 'parallel', planar, 'normal","authors":"Line","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"line, 'direction', 'orthogonal', planar, 'normal') 7. (planar, 'location', 'co-planar', planar, 'location') 8. (planar, 'normal', 'parallel', planar, 'normal')","title":"direction', 'orthogonal', planar, 'normal') 7. (planar, 'location', 'co-planar', planar, 'location') 8. (planar, 'normal', 'parallel', planar, 'normal","authors":"Line","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"planar, 'normal', 'orthogonal', planar, 'normal') 10. (planar, 'normal', 'parallel', texture, 'normal')","title":"planar, 'normal', 'orthogonal', planar, 'normal') 10. (planar, 'normal', 'parallel', texture","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"planar, 'normal', 'orthogonal', texture, 'normal')","title":"planar, 'normal', 'orthogonal', texture","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"planar, 'location', 'contact', vehicle, 'cubiod')","title":"location', 'contact', vehicle, 'cubiod","authors":"Planar","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"planar, 'location', 'contact', human, 'cubiod')","title":"location', 'contact', human, 'cubiod","authors":"Planar","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"planar, 'normal', 'parallel', human, 'direction') 15. (texture, 'location', 'contact', texture, 'location')","title":"normal', 'parallel', human, 'direction') 15. (texture, 'location', 'contact', texture","authors":"Planar","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"vehicle, 'length', '=', vehicle, 'length')","title":"vehicle, 'length","authors":"Vehicle","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"vehicle, 'width', '=', vehicle, 'width')","title":"vehicle, 'width","authors":"Vehicle","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"vehicle, 'height', '=', vehicle, 'height')","title":"vehicle, 'height","authors":"Vehicle","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"vehicle, 'height', 'parallel', vehicle, 'orientation')","title":"height', 'parallel', vehicle, 'orientation","authors":"Vehicle","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"vehicle, 'length', '>', human, 'height')","title":"length","authors":"Vehicle","year":null,"venue":"human, 'height","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"human, 'height', '=', human, 'height') References","title":"human, 'height', '=', human, 'height') References","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Criminisi and Zisserman, 2000] Antonio Criminisi and Andrew Zisserman. Shape from texture: Homogeneity revisited. In B- MVC, pages 1-10, 2000.","title":"Shape from texture: Homogeneity revisited","authors":"Zisserman Criminisi, Antonio Criminisi, Andrew Zisserman","year":"2000","venue":"B-MVC","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Davis and Marcus, 2015] Ernest Davis and Gary Marcus. Com- monsense reasoning and commonsense knowledge in artificial intelligence. Communications of the ACM, 58(9):92-103, 2015.","title":"Commonsense reasoning and commonsense knowledge in artificial intelligence","authors":"Marcus ; Davis, Ernest Davis, Gary Marcus","year":"2015","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Davis et al., 1993] Randall Davis, Howard Shrobe, and Peter S- zolovits. What is a knowledge representation? AI magazine, 14(1):17, 1993.","title":"What is a knowledge representation?","authors":"Randall Davis, Howard Shrobe, Peter Szolovits","year":"1993","venue":"AI magazine","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Del Pero et al., 2011] Luca Del Pero, Jinyan Guan, Ernesto Brau, Joseph Schlecht, and Kobus Barnard. Sampling bedrooms. In CVPR, pages 2009-2016. IEEE, 2011.","title":"Sampling bedrooms","authors":"Luca Del Pero, Jinyan Guan, Ernesto Brau, Joseph Schlecht, Kobus Barnard","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Del Pero et al., 2013] Luca Del Pero, Joshua Bowdish, Bonnie K- ermgard, Emily Hartley, and Kobus Barnard. Understanding bayesian rooms using composite 3d object models. In CVPR, pages 153-160, 2013.","title":"Understanding bayesian rooms using composite 3d object models","authors":"Luca Del Pero, Joshua Bowdish, Bonnie Kermgard, Emily Hartley, Kobus Barnard","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[del Rincón et al., 2013] Jesús Martínez del Rincón, Maria J Santofimia, and Jean-Christophe Nebel. Common-sense reason- ing for human action recognition. Pattern Recognition Letters, 34(15):1849-1860, 2013.","title":"Jesús Martínez del Rincón, Maria J Santofimia, and Jean-Christophe Nebel. Common-sense reasoning for human action recognition","authors":"Del Rincón","year":"2013","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Dellaert et al., 2000] Frank Dellaert, Steven M Seitz, Charles E Thorpe, and Sebastian Thrun. Structure from motion without cor- respondence. In CVPR, volume 2, pages 557-564. IEEE, 2000.","title":"Structure from motion without correspondence","authors":"Frank Dellaert, Steven M Seitz, Charles E Thorpe, Sebastian Thrun","year":"2000","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Everingham et al., 2015] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zis- serman. The pascal visual object classes challenge: A retrospec- tive. IJCV, 111(1):98-136, 2015.","title":"The pascal visual object classes challenge: A retrospective","authors":"Mark Everingham, Ali Sm, Luc Eslami, Van Gool, K I Christopher, John Williams, Andrew Winn, Zisserman","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Felzenszwalb et al., 2010] Pedro F Felzenszwalb, Ross B Gir- shick, David McAllester, and Deva Ramanan. Object detec- tion with discriminatively trained part-based models. TPAMI, 32(9):1627-1645, 2010.","title":"Object detection with discriminatively trained part-based models","authors":"Pedro F Felzenszwalb, Ross B Girshick, David Mcallester, Deva Ramanan","year":"2010","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Fouhey et al., 2014] David F Fouhey, Vincent Delaitre, Abhinav Gupta, Alexei A Efros, Ivan Laptev, and Josef Sivic. People watching: Human actions as a cue for single view geometry. I- JCV, 110(3):259-274, 2014.","title":"People watching: Human actions as a cue for single view geometry","authors":"Vincent David F Fouhey, Abhinav Delaitre, Alexei A Gupta, Ivan Efros, Josef Laptev, Sivic","year":"2014","venue":"I-JCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Furukawa et al., 2009] Yasutaka Furukawa, Brian Curless, Steven M Seitz, and Richard Szeliski. Manhattan-world stereo. In CVPR, pages 1422-1429. IEEE, 2009.","title":"Manhattan-world stereo","authors":"Yasutaka Furukawa, Brian Curless, Steven M Seitz, Richard Szeliski","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Grabner et al., 2011] Helmut Grabner, Juergen Gall, and Luc Van Gool. What makes a chair a chair? In CVPR, pages 1529- 1536. IEEE, 2011.","title":"What makes a chair a chair?","authors":"Helmut Grabner, Juergen Gall, Luc Van Gool","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Gupta et al., 2010] Abhinav Gupta, Alexei Efros, and Martial Hebert. Blocks world revisited: Image understanding using qual- itative geometry and mechanics. ECCV, pages 482-496, 2010.","title":"Blocks world revisited: Image understanding using qualitative geometry and mechanics","authors":"Abhinav Gupta, Alexei Efros, Martial Hebert","year":"2010","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Han and Zhu, 2009] Feng Han and Song-Chun Zhu. Bottom- up/top-down image parsing with attribute grammar. TPAMI, 31(1):59-73, 2009.","title":"Bottomup/top-down image parsing with attribute grammar","authors":"Zhu ; ] Feng Han, Song-Chun Han, Zhu","year":"2009","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Heitz et al., 2009] Geremy Heitz, Stephen Gould, Ashutosh Sax- ena, and Daphne Koller. Cascaded classification models: Com- bining models for holistic scene understanding. In Advances in Neural Information Processing Systems, pages 641-648, 2009.","title":"Cascaded classification models: Combining models for holistic scene understanding","authors":"Geremy Heitz, Stephen Gould, Ashutosh Saxena, Daphne Koller","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Hoiem et al., 2005] Derek Hoiem, Alexei A Efros, and Martial Hebert. Geometric context from a single image. In ICCV, vol- ume 1, pages 654-661. IEEE, 2005.","title":"Geometric context from a single image","authors":"Derek Hoiem, Alexei A Efros, Martial Hebert","year":"2005","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Hoiem et al., 2008] Derek Hoiem, Alexei A Efros, and Martial Hebert. Closing the loop in scene interpretation. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Con- ference on, pages 1-8. IEEE, 2008.","title":"Closing the loop in scene interpretation","authors":"Derek Hoiem, Alexei A Efros, Martial Hebert","year":"2008","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Koutsourakis et al., 2009] Panagiotis Koutsourakis, Loic Simon, Olivier Teboul, Georgios Tziritas, and Nikos Paragios. Single view reconstruction using shape grammars for urban environ- ments. In ICCV, pages 1795-1802. IEEE, 2009.","title":"Single view reconstruction using shape grammars for urban environments","authors":"Panagiotis Koutsourakis, Loic Simon, Olivier Teboul, Georgios Tziritas, Nikos Paragios","year":"2009","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Li et al., 2012] Bo Li, Kun Peng, Xianghua Ying, and Hongbin Zha. Vanishing point detection using cascaded 1d hough trans- form from single images. Pattern Recognition Letters, 33(1):1-8, 2012.","title":"Vanishing point detection using cascaded 1d hough transform from single images","authors":"Bo Li, Kun Peng, Xianghua Ying, Hongbin Zha","year":"2012","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"Liu et al., ] Xiaobai Liu, Yadong Mu, and Liang Lin. A stochastic image grammar for fine-grained 3d scene reconstruction.","title":"A stochastic image grammar for fine-grained 3d scene reconstruction","authors":"Liu","year":null,"venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2016/484"},{"position":38,"raw":"[Liu et al., 2014] Xiaobai Liu, Yibiao Zhao, and Song-Chun Zhu. Single-view 3d scene parsing by attributed grammar. In CVPR, pages 684-691, 2014.","title":"Single-view 3d scene parsing by attributed grammar","authors":"Xiaobai Liu, Yibiao Zhao, Song-Chun Zhu","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Lucas et al., 1981] Bruce D Lucas, Takeo Kanade, et al. An iter- ative image registration technique with an application to stereo vision. 1981.","title":"An iterative image registration technique with an application to stereo vision","authors":"Takeo Bruce D Lucas, Kanade","year":"1981","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/1981/127"},{"position":40,"raw":"[Mottaghi et al., 2016] Roozbeh Mottaghi, Sanja Fidler, Alan Yuille, Raquel Urtasun, and Devi Parikh. Human-machine crf- s for identifying bottlenecks in scene understanding. TPAMI, 38(1):74-87, 2016.","title":"Human-machine crfs for identifying bottlenecks in scene understanding","authors":"Roozbeh Mottaghi, Sanja Fidler, Alan Yuille, Raquel Urtasun, Devi Parikh","year":"2016","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Nayar and Nakagawa, 1990] Shree K Nayar and Yasuo Nakagawa. Shape from focus: An effective approach for rough surfaces. In ICRA, pages 218-225. IEEE, 1990.","title":"Shape from focus: An effective approach for rough surfaces","authors":"Nakagawa ; Nayar, K Shree, Yasuo Nayar, Nakagawa","year":"1990","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Ren and Malik, 2003] Xiaofeng Ren and Jitendra Malik. Learn- ing a classification model for segmentation. In ICCV, volume 1, pages 10-17, 2003.","title":"Learning a classification model for segmentation","authors":"Xiaofeng Ren, Jitendra Malik","year":"2003","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Toppe et al., 2013] Eno Toppe, Claudia Nieuwenhuis, and Daniel Cremers. Relative volume constraints for single view 3d recon- struction. In CVPR, pages 177-184, 2013.","title":"Relative volume constraints for single view 3d reconstruction","authors":"Toppe","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Tu and Zhu, 2002] Zhuowen Tu and Song-Chun Zhu. Image seg- mentation by data-driven markov chain monte carlo. TPAMI, 24(5):657-673, 2002.","title":"Zhuowen Tu and Song-Chun Zhu. Image segmentation by data-driven markov chain monte carlo","authors":"Zhu Tu","year":"2002","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Wang et al., 2007] Shiaokai Wang, William Pentney, Ana-Maria Popescu, Tanzeem Choudhury, and Matthai Philipose. Common sense based joint training of human activity recognizers. In IJ- CAI, volume 7, pages 2237-2242, 2007.","title":"Common sense based joint training of human activity recognizers","authors":"Wang","year":"2007","venue":"IJ-CAI","doi":null,"url":null,"internal_url":"/proceedings/2007/360"},{"position":46,"raw":"[Wang et al., 2013] Shuo Wang, Jungseock Joo, Yizhou Wang, and Song-Chun Zhu. Weakly supervised learning for attribute local- ization in outdoor scenes. In CVPR, pages 3111-3118, 2013.","title":"Weakly supervised learning for attribute localization in outdoor scenes","authors":"Shuo Wang, Jungseock Joo, Yizhou Wang, Song-Chun Zhu","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Wang et al., 2015a] Shenlong Wang, Sanja Fidler, and Raquel Ur- tasun. Holistic 3d scene understanding from a single geo-tagged image. In CVPR, pages 3964-3972, 2015.","title":"Holistic 3d scene understanding from a single geo-tagged image","authors":"Wang","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Wang et al., 2015b] Shenlong Wang, Sanja Fidler, and Raquel Ur- tasun. Lost shopping! monocular localization in large indoor spaces. In ICCV, pages 2695-2703, 2015.","title":"Lost shopping! monocular localization in large indoor spaces","authors":"Wang","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Wyatt et al., 2005] Danny Wyatt, Matthai Philipose, and Tanzeem Choudhury. Unsupervised activity recognition using automati- cally mined common sense. In AAAI, volume 5, pages 21-27, 2005.","title":"Unsupervised activity recognition using automatically mined common sense","authors":"Danny Wyatt, Matthai Philipose, Tanzeem Choudhury","year":"2005","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zhang et al., 1999] Ruo Zhang, Ping-Sing Tsai, James Edwin Cry- er, and Mubarak Shah. Shape-from-shading: a survey. TPAMI, 21(8):690-706, 1999.","title":"Shape-from-shading: a survey","authors":"Ruo Zhang, Ping-Sing Tsai, James Edwin Cryer, Mubarak Shah","year":"1999","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Zhao and Zhu, 2011] Yibiao Zhao and Song-Chun Zhu. Image parsing with stochastic scene grammar. In Advances in Neural Information Processing Systems, pages 73-81, 2011.","title":"Image parsing with stochastic scene grammar","authors":"Zhu Zhao, Yibiao Zhao, Song-Chun Zhu","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Zhu et al., 2014] Yuke Zhu, Alireza Fathi, and Li Fei-Fei. Reason- ing about object affordances in a knowledge base representation. In ECCV, pages 408-424. Springer, 2014.","title":"Reasoning about object affordances in a knowledge base representation","authors":"Yuke Zhu, Alireza Fathi, Li Fei-Fei","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Zhu et al., 2015] Yixin Zhu, Yibiao Zhao, and Song Chun Zhu. Understanding tools: Task-oriented object modeling, learning and recognition. In CVPR, pages 2855-2864, 2015.","title":"Understanding tools: Task-oriented object modeling, learning and recognition","authors":"Yixin Zhu, Yibiao Zhao, Song Chun, Zhu","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Zitnick and Parikh, 2013] C Lawrence Zitnick and Devi Parikh. Bringing semantics into focus using visual abstraction. In CVPR, pages 3009-3016, 2013.","title":"Bringing semantics into focus using visual abstraction","authors":"Lawrence Zitnick, Devi Parikh","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}