{"type":"ScholarlyArticle","title":"A Unifying View of Geometry, Semantics, and Data Association in SLAM","authors":[{"name":"Nikolay Atanasov","slug":"nikolay-atanasov","given":"Nikolay","family":"Atanasov","affiliation":"Electrical and Computer Engineering , University of California , San Diego , CA 92093 , USA"},{"name":"Sean L. Bowman","slug":"sean-l-bowman","given":"Sean L.","family":"Bowman","affiliation":"GRASP Laboratory , University of Pennsylvania , PA 19104 , USA"},{"name":"Kostas Daniilidis","slug":"kostas-daniilidis","given":"Kostas","family":"Daniilidis","affiliation":"GRASP Laboratory , University of Pennsylvania , PA 19104 , USA"},{"name":"George J. Pappas","slug":"george-j-pappas","given":"George J.","family":"Pappas","affiliation":"GRASP Laboratory , University of Pennsylvania , PA 19104 , USA"}],"abstract":"Traditional approaches for simultaneous localization and mapping (SLAM) rely on geometric features such as points, lines, and planes to infer the environment structure. They make hard decisions about the (data) association between observed features and mapped landmarks to update the environment model. This paper makes two contributions to the state of the art in SLAM. First, it generalizes the purely geometric model by introducing semantically meaningful objects, represented as structured models of mid-level part features. Second, instead of making hard, potentially wrong associations between semantic features and objects, it shows that SLAM inference can be performed efficiently with probabilistic data association. The approach not only allows building meaningful maps (containing doors, chairs, cars, etc.) but also offers significant advantages in ambiguous environments.","doi":"10.24963/ijcai.2018/722","url":"https://ijcai.org/proceedings/2018/722","pdf":"https://ijcai.org/proceedings/2018/0722.pdf","markdown":"https://ijcai.org/proceedings/2018/722.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/722","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Sister Conferences Best Papers","track":"Best Sister Conferences","number":722,"pages":{"first":5204,"last":5208},"keywords":["Robotics: Localization, Mapping, State Estimation","Robotics: Robotics and Vision","Computer Vision: 2D and 3D Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Atanasov et al., 2015] N. Atanasov, M. Zhu, K. Daniilidis, and G. Pappas. Localization from semantic observations via the ma- trix permanent. IJRR, 35:73-99, 2015.","title":"Localization from semantic observations via the matrix permanent","authors":"N Atanasov, M Zhu, K Daniilidis, G Pappas","year":"2015","venue":"IJRR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bao and Savarese, 2011] S. Bao and S. Savarese. Semantic Struc- ture from Motion. In IEEE CVPR, pages 2025-2032, 2011.","title":"Semantic Structure from Motion","authors":"S Bao, S Savarese","year":"2011","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bloesch et al., 2015] M. Bloesch, S. Omari, M. Hutter, and R. Siegwart. Robust visual inertial odometry using a direct EKF- based approach. In IEEE/RSJ IROS, pages 298-304, 2015.","title":"Robust visual inertial odometry using a direct EKFbased approach","authors":"M Bloesch, S Omari, M Hutter, R Siegwart","year":"2015","venue":"IEEE/RSJ IROS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bowman et al., 2017] S. Bowman, N. Atanasov, K. Daniilidis, and G. Pappas. Probabilistic Data Association for Semantic SLAM. In IEEE ICRA, 2017. (Best Conference Paper Award).","title":"Probabilistic Data Association for Semantic SLAM","authors":"S Bowman, N Atanasov, K Daniilidis, G Pappas","year":"2017","venue":"IEEE ICRA","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cadena et al., 2016] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J. Leonard. Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age. IEEE T-RO, 32(6):1309-1332, 2016.","title":"Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age","authors":"C Cadena, L Carlone, H Carrillo, Y Latif, D Scaramuzza, J Neira, I Reid, J Leonard","year":"2016","venue":"IEEE T-RO","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dubout and Fleuret, 2013] C. Dubout and F. Fleuret. Deformable part models with individual part scaling. In BMVC, 2013.","title":"Deformable part models with individual part scaling","authors":"C Dubout, F Fleuret","year":"2013","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Engel et al., 2014] J. Engel, T. Schöps, and D. Cremers. LSD- SLAM: Large-scale direct monocular SLAM. In ECCV, 2014.","title":"LSD-SLAM: Large-scale direct monocular SLAM","authors":"J Engel, T Schöps, D Cremers","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Felzenszwalb et al., 2010] P. Felzenszwalb, R. Girshick, D. McAllester, and D. Ramanan. Object Detection with Discriminatively Trained Part-Based Models. IEEE Trans. on PAMI, 32(9):1627-1645, 2010.","title":"Object Detection with Discriminatively Trained Part-Based Models","authors":"P Felzenszwalb, R Girshick, D Mcallester, D Ramanan","year":"2010","venue":"IEEE Trans. on PAMI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Forster et al., 2015] C. Forster, L. Carlone, F. Dellaert, and D. Scaramuzza. IMU Preintegration on Manifold for Efficient Visual-Inertial Maximum-a-Posteriori Estimation. In Robotics: Science and Systems, 2015.","title":"IMU Preintegration on Manifold for Efficient Visual-Inertial Maximum-a-Posteriori Estimation","authors":"C Forster, L Carlone, F Dellaert, D Scaramuzza","year":"2015","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Forster et al., 2016] C. Forster, Z. Zhang, M. Gassner, M. Werl- berger, and D. Scaramuzza. SVO: Semi-Direct Visual Odometry for Monocular and Multi-Camera Systems. IEEE T-RO, 2016.","title":"SVO: Semi-Direct Visual Odometry for Monocular and Multi-Camera Systems","authors":"C Forster, Z Zhang, M Gassner, M Werlberger, D Scaramuzza","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gálvez-López et al., 2016] D. Gálvez-López, M. Salas, J. Tardós, and J. Montiel. Real-time monocular object SLAM. Robotics and Autonomous Systems, 75:435-449, 2016.","title":"Real-time monocular object SLAM","authors":"D Gálvez-López, M Salas, J Tardós, J Montiel","year":"2016","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Geiger et al., 2011] A. Geiger, J. Ziegler, and C. Stiller. Stere- oScan: Dense 3d Reconstruction in Real-time. In Intelligent Ve- hicles Symposium (IV), pages 963-968, 2011.","title":"Stere-oScan: Dense 3d Reconstruction in Real-time","authors":"A Geiger, J Ziegler, C Stiller","year":"2011","venue":"Intelligent Vehicles Symposium (IV)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Geiger et al., 2012] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In IEEE CVPR, 2012.","title":"Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite","authors":"A Geiger, P Lenz, R Urtasun","year":"2012","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Girshick et al., 2014] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In IEEE CVPR, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"R Girshick, J Donahue, T Darrell, J Malik","year":"2014","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[He et al., 2016] K. He, X. Zhang, S. Ren, and J. Sun. Deep resid- ual learning for image recognition. In IEEE CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"K He, X Zhang, S Ren, J Sun","year":"2016","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hornung et al., 2013] A. Hornung, K. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard. OctoMap: An efficient proba- bilistic 3D mapping framework based on octrees. Autonomous Robots, 34(3):189-206, 2013.","title":"OctoMap: An efficient probabilistic 3D mapping framework based on octrees","authors":"A Hornung, K Wurm, M Bennewitz, C Stachniss, W Burgard","year":"2013","venue":"Autonomous Robots","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jerrum et al., 2004] M. Jerrum, A. Sinclair, and E. Vigoda. A Polynomial-time Approximation Algorithm for the Permanent of a Matrix with Nonnegative Entries. Journal of the ACM, 51(4):671-697, 2004.","title":"A Polynomial-time Approximation Algorithm for the Permanent of a Matrix with Nonnegative Entries","authors":"M Jerrum, A Sinclair, E Vigoda","year":"2004","venue":"Journal of the ACM","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kabsch, 1978] W. Kabsch. A discussion of the solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A, 34(5):827-828, 1978.","title":"A discussion of the solution for the best rotation to relate two sets of vectors","authors":"W Kabsch","year":"1978","venue":"Acta Crystallographica Section A","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kaess et al., 2008] M. Kaess, A. Ranganathan, and F. Dellaert. iSAM: Incremental Smoothing and Mapping. IEEE T-RO, 24(6):1365-1378, 2008.","title":"iSAM: Incremental Smoothing and Mapping","authors":"M Kaess, A Ranganathan, F Dellaert","year":"2008","venue":"IEEE T-RO","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kaess et al., 2012] M. Kaess, H. Johannsson, R. Roberts, V. Ila, J. Leonard, and F. Dellaert. iSAM2: Incremental Smoothing and Mapping Using the Bayes Tree. IJRR, 31(2):216-235, 2012.","title":"iSAM2: Incremental Smoothing and Mapping Using the Bayes Tree","authors":"M Kaess, H Johannsson, R Roberts, V Ila, J Leonard, F Dellaert","year":"2012","venue":"IJRR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Koller and Friedman, 2009] D. Koller and N. Friedman. Proba- bilistic Graphical Models. MIT press, 2009.","title":"Probabilistic Graphical Models","authors":"Friedman Koller, D Koller, N Friedman","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kostavelis and Gasteratos, 2015] Io. Kostavelis and A. Gasteratos. Semantic mapping for mobile robotics tasks: A survey. Robotics and Autonomous Systems, 66:86-103, 2015.","title":"Semantic mapping for mobile robotics tasks: A survey","authors":"Io, A Kostavelis, Gasteratos","year":"2015","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Krizhevsky et al., 2012] A. Krizhevsky, I. Sutskever, and Geoffrey H. ImageNet Classification with Deep Convolutional Neural Net- works. In NIPS, pages 1097-1105. 2012.","title":"ImageNet Classification with Deep Convolutional Neural Networks","authors":"A Krizhevsky, I Sutskever, Geoffrey H","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kümmerle et al., 2011] R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard. g2o: A General Framework for Graph Optimization. In IEEE ICRA, pages 3607-3613, 2011.","title":"g2o: A General Framework for Graph Optimization","authors":"R Kümmerle, G Grisetti, H Strasdat, K Konolige, W Burgard","year":"2011","venue":"IEEE ICRA","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ma et al., 2012] Y. Ma, S. Soatto, J. Kosecka, and S. Sastry. An Invitation to 3-D Vision: from Images to Geometric Models, vol- ume 26. Springer Science & Business Media, 2012.","title":"An Invitation to 3-D Vision: from Images to Geometric Models","authors":"Y Ma, S Soatto, J Kosecka, S Sastry","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mourikis and Roumeliotis, 2006] A. Mourikis and S. Roumeliotis. A Multi-State Constraint Kalman Filter for Vision-aided Inertial Navigation. Technical report, University of Minnesota, 2006.","title":"A Multi-State Constraint Kalman Filter for Vision-aided Inertial Navigation","authors":"A Mourikis, S Roumeliotis","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Munkres, 1957] J. Munkres. Algorithms for the Assignment and Transportation Problems. Journal of SIAM, 5(1):32-38, 1957.","title":"Algorithms for the Assignment and Transportation Problems","authors":"J Munkres","year":"1957","venue":"Journal of SIAM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Mur-Artal and Tardós, 2017] R. Mur-Artal and J. Tardós. ORB- SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras. IEEE T-RO, 33(5):1255-1262, 2017.","title":"ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras","authors":"R Mur-Artal, J Tardós","year":"2017","venue":"IEEE T-RO","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Murthy et al., 2017] J. Murthy, S. Sharma, and K. Krishna. Shape priors for real-time monocular object localization in dynamic en- vironments. In IEEE/RSJ IROS, pages 1768-1774, 2017.","title":"Shape priors for real-time monocular object localization in dynamic environments","authors":"J Murthy, S Sharma, K Krishna","year":"2017","venue":"IEEE/RSJ IROS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Neira and Tardós, 2001] J. Neira and J. Tardós. Data Association in Stochastic Mapping Using the Joint Compatibility Test. IEEE T-RO, 17(6):890-897, 2001.","title":"Data Association in Stochastic Mapping Using the Joint Compatibility Test","authors":"J Neira, J Tardós","year":"2001","venue":"IEEE T-RO","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Pavlakos et al., 2017] G. Pavlakos, X. Zhou, A. Chan, K. G. Der- panis, and K. Daniilidis. 6-DoF object pose from semantic key- points. In IEEE ICRA, pages 2011-2018, 2017.","title":"6-DoF object pose from semantic keypoints","authors":"G Pavlakos, X Zhou, A Chan, K G Derpanis, K Daniilidis","year":"2017","venue":"IEEE ICRA","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Qin et al., 2017] T. Qin, P. Li, and S. Shen. VINS-Mono: A robust and versatile monocular visual-inertial state estimator. In arXiv preprint:1708.03852, 2017.","title":"VINS-Mono: A robust and versatile monocular visual-inertial state estimator","authors":"T Qin, P Li, S Shen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Redmon and Farhadi, 2016] J. Redmon and A. Farhadi. Yolo9000: Better, faster, stronger. arXiv preprint:1612.08242, 2016.","title":"Yolo","authors":"J Redmon, A Farhadi","year":"2016","venue":"Better, faster, stronger","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Ren et al., 2015] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Pro- posal Networks. In NIPS, pages 91-99. 2015.","title":"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks","authors":"S Ren, K He, R Girshick, J Sun","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Rublee et al., 2011] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski. ORB: An efficient alternative to SIFT or SURF. In ICCV, pages 2564-2571, 2011.","title":"ORB: An efficient alternative to SIFT or SURF","authors":"E Rublee, V Rabaud, K Konolige, G Bradski","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Salas-Moreno et al., 2013] R. Salas-Moreno, R. Newcombe, H. Strasdat, P. Kelly, and A. Davison. SLAM++: Simultaneous Localisation and Mapping at the Level of Objects. In IEEE CVPR, 2013.","title":"SLAM++: Simultaneous Localisation and Mapping at the Level of Objects","authors":"R Salas-Moreno, R Newcombe, H Strasdat, P Kelly, A Davison","year":"2013","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Simonyan and Zisserman, 2014] K. Simonyan and A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556, 2014.","title":"Very Deep Convolutional Networks for Large-Scale Image Recognition","authors":"K Simonyan, A Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Strasdat et al., 2010] H. Strasdat, J. Montiel, and A. Davison. Real-time monocular SLAM: Why filter? In IEEE ICRA, 2010.","title":"Real-time monocular SLAM: Why filter?","authors":"H Strasdat, J Montiel, A Davison","year":"2010","venue":"IEEE ICRA","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Thrun et al., 2005] S. Thrun, W. Burgard, and D. Fox. Probabilis- tic Robotics. MIT Press Cambridge, 2005.","title":"Probabilistic Robotics","authors":"S Thrun, W Burgard, D Fox","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Whelan et al., 2016] T. Whelan, R. Salas-Moreno, B., A. Davison, and S. Leutenegger. ElasticFusion: Real-Time Dense SLAM and Light Source Estimation. IJRR, 35(14):1697-1716, 2016.","title":"ElasticFusion: Real-Time Dense SLAM and Light Source Estimation","authors":"T Whelan, R Salas-Moreno, B, A Davison, S Leutenegger","year":"2016","venue":"IJRR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}