{"type":"ScholarlyArticle","title":"PlaceNet: Neural Spatial Representation Learning with Multimodal Attention","authors":[{"name":"Chung-Yeon Lee","slug":"chung-yeon-lee","given":"Chung-Yeon","family":"Lee","affiliation":"Seoul National University , 2 Surromind ,"},{"name":"Youngjae Yoo","slug":"youngjae-yoo","given":"Youngjae","family":"Yoo","affiliation":"Seoul National University , 2 Surromind ,"},{"name":"Byoung-Tak Zhang","slug":"byoung-tak-zhang","given":"Byoung-Tak","family":"Zhang","affiliation":"Seoul National University , 2 Surromind ,; AIIS"}],"abstract":"Spatial representation capable of learning a myriad of environmental features is a significant challenge for natural spatial understanding of mobile AI agents.\r\nDeep generative models have the potential of discovering rich representations of observed 3D scenes.\r\nHowever, previous approaches have been mainly evaluated on simple environments, or focused only on high-resolution rendering of small-scale scenes, hampering generalization of the representations to various spatial variability.\r\nTo address this, we present PlaceNet, a neural representation that learns through random observations in a self-supervised manner, and represents observed scenes with triplet attention using visual, topographic, and semantic cues.\r\nWe evaluate the proposed method on a large-scale multimodal scene dataset consisting of 120 million indoor scenes, and show that PlaceNet successfully generalizes to various environments with lower training loss, higher image quality and structural similarity of predicted scenes, compared to a competitive baseline model.\r\nAdditionally, analyses of the representations demonstrate that PlaceNet activates more specialized and larger numbers of kernels in the spatial representation, capturing multimodal spatial properties in complex environments.","doi":"10.24963/ijcai.2022/144","url":"https://ijcai.org/proceedings/2022/144","pdf":"https://ijcai.org/proceedings/2022/0144.pdf","markdown":"https://ijcai.org/proceedings/2022/144.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/144","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":144,"pages":{"first":1031,"last":1038},"keywords":["Robotics: Perception","Humans and AI: Cognitive Systems","Computer Vision: Scene analysis and understanding","Machine Learning: Multi-modal learning","Machine Learning: Representation learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986793","length_sec":68,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984936","length_sec":355,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bailey and Durrant-Whyte, 2006] Tim Bailey and Hugh Durrant-Whyte. Simultaneous localization and mapping (SLAM): Part II. IEEE Robotics & Automation Magazine, 13(3):108-117, 2006.","title":"Simultaneous localization and mapping (SLAM): Part II","authors":"Durrant-Whyte ; Bailey, Tim Bailey, Hugh Durrant-Whyte","year":"2006","venue":"IEEE Robotics & Automation Magazine","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cadena et al., 2016] Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, Ian Reid, and John J Leonard. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on Robotics, 32(6):1309-1332, 2016.","title":"Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age","authors":"Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, Ian Reid, John J Leonard","year":"2016","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Williams, 1993] Shenchang Eric Chen and Lance Williams. View interpolation for image synthesis. In Pro- ceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pages 279-288, 1993.","title":"View interpolation for image synthesis","authors":"Chen, Williams ;, Eric Chen, Lance Williams","year":"1993","venue":"Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2017] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.","title":"Rethinking atrous convolution for semantic image segmentation","authors":"Liang-Chieh Chen, George Papandreou, Florian Schroff, Hartwig Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Eslami et al., 2018] SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene represen- tation and rendering. Science, 360(6394):1204-1210, 2018.","title":"Neural scene representation and rendering","authors":"Eslami","year":"2018","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu et al., 2019] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual at- tention network for scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 3146-3154, 2019.","title":"Dual attention network for scene segmentation","authors":"Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, Hanqing Lu","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gregor et al., 2015] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. DRAW: a recurrent neural network for image generation. In Proceedings of the International Conference on Ma- chine Learning (ICML), volume 37, pages 1462-1471, 2015.","title":"DRAW: a recurrent neural network for image generation","authors":"Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra","year":"2015","venue":"Proceedings of the International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gupta et al., 2017] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2616-2625, 2017.","title":"Cognitive mapping and planning for visual navigation","authors":"Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, Jitendra Malik","year":"2017","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Herweg and Kahana, 2018] Nora A Herweg and Michael J Kahana. Spatial representations in the human brain. Fron- tiers in Human Neuroscience, 12:297, 2018.","title":"Spatial representations in the human brain","authors":"Kahana Herweg, Nora A Herweg, Michael J Kahana","year":"2018","venue":"Frontiers in Human Neuroscience","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mildenhall et al., 2020] Ben Mildenhall, Pratul P Srini- vasan, Matthew Tancik, Jonathan T Barron, Ravi Ra- mamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), pages 405-421. Springer, 2020.","title":"NeRF: Representing scenes as neural radiance fields for view synthesis","authors":"Ben Mildenhall, P Pratul, Matthew Srinivasan, Jonathan T Tancik, Ravi Barron, Ren Ramamoorthi, Ng","year":"2020","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Moser et al., 2008] Edvard I Moser, Emilio Kropff, and May-Britt Moser. Place cells, grid cells, and the brain's spatial representation system. Annu. Rev. Neurosci., 31:69-89, 2008.","title":"Edvard I Moser, Emilio Kropff, and May-Britt Moser. Place cells, grid cells, and the brain's spatial representation system","authors":"Moser","year":"2008","venue":"Annu. Rev. Neurosci","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[O'Keefe, 1976] John O'Keefe. Place units in the hippocam- pus of the freely moving rat. Experimental Neurology, 51(1):78-109, 1976.","title":"Place units in the hippocampus of the freely moving rat","authors":"O' O'keefe ; John, Keefe","year":"1976","venue":"Experimental Neurology","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Poudel et al., 2019] Rudra PK Poudel, Stephan Liwicki, and Roberto Cipolla. Fast-SCNN: fast semantic segmen- tation network. arXiv preprint arXiv:1902.04502, 2019.","title":"Fast-SCNN: fast semantic segmentation network","authors":"P K Rudra, Stephan Poudel, Roberto Liwicki, Cipolla","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rosenbaum et al., 2018] Dan Rosenbaum, Frederic Besse, Fabio Viola, Danilo J Rezende, and SM Eslami. Learning models for visual 3D localization with implicit mapping. arXiv preprint arXiv:1807.03149, 2018.","title":"Learning models for visual 3D localization with implicit mapping","authors":"Dan Rosenbaum, Frederic Besse, Fabio Viola, Danilo J Rezende, Eslami","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sitzmann et al., 2019] Vincent Sitzmann, Justus Thies, Fe- lix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. DeepVoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2437-2446, 2019.","title":"DeepVoxels: Learning persistent 3d feature embeddings","authors":"Justus Vincent Sitzmann, Felix Thies, Matthias Heide, Gordon Nießner, Michael Wetzstein, Zollhofer","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Song et al., 2017] Shuran Song, Fisher Yu, Andy Zeng, An- gel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1746- 1754, 2017.","title":"Semantic scene completion from a single depth image","authors":"Song","year":"2017","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tewari et al., 2020] Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, et al. State of the art on neu- ral rendering. Computer Graphics Forum, 39(2):701-727, 2020.","title":"State of the art on neural rendering","authors":"Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner","year":"2020","venue":"Computer Graphics Forum","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tobin et al., 2019] Joshua Tobin, Wojciech Zaremba, and Pieter Abbeel. Geometry-aware neural rendering. In Ad- vances in Neural Information Processing Systems (NIPS), pages 11559-11569, 2019.","title":"Geometry-aware neural rendering","authors":"Joshua Tobin, Wojciech Zaremba, Pieter Abbeel","year":"2019","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans- actions on Image Processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2018] Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3D environment. arXiv preprint arXiv:1801.02209, 2018.","title":"Building generalizable agents with a realistic and rich 3D environment","authors":"Yi Wu, Yuxin Wu, Georgia Gkioxari, Yuandong Tian","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xingjian et al., 2015] Shi Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional LSTM network: A machine learn- ing approach for precipitation nowcasting. In Advances in Neural Information Processing Systems (NIPS), pages 802-810, 2015.","title":"Convolutional LSTM network: A machine learning approach for precipitation nowcasting","authors":"Zhourong Shi Xingjian, Hao Chen, Dit-Yan Wang, Wai-Kin Yeung, Wang-Chun Wong, Woo","year":"2015","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yamamoto et al., 2018] Takashi Yamamoto, Koji Terada, Akiyoshi Ochiai, Fuminori Saito, Yoshiaki Asahara, and Kazuto Murase. Development of the research platform of a domestic mobile manipulator utilized for international competition and field test. In International Conference on Intelligent Robots and Systems (IROS), pages 7675-7682, 2018.","title":"Development of the research platform of a domestic mobile manipulator utilized for international competition and field test","authors":"Takashi Yamamoto, Koji Terada, Akiyoshi Ochiai, Fuminori Saito, Yoshiaki Asahara, Kazuto Murase","year":"2018","venue":"International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2016] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros. View synthesis by appearance flow. In Proceedings of the European Conference on Computer Vision (ECCV), pages 286-301. Springer, 2016.","title":"View synthesis by appearance flow","authors":"Zhou","year":"2016","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 633-641, 2017.","title":"Scene parsing through ade20k dataset","authors":"Zhou","year":"2017","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2019] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302-321, 2019.","title":"Semantic understanding of scenes through the ade20k dataset","authors":"Zhou","year":"2019","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}