{"type":"ScholarlyArticle","title":"Translating Images into Maps (Extended Abstract)","authors":[{"name":"Avishkar Saha","slug":"avishkar-saha","given":"Avishkar","family":"Saha","affiliation":"Centre for Vision Speech and Signal Processing , University of Surrey , Guildford , UK"},{"name":"Oscar Mendez","slug":"oscar-mendez","given":"Oscar","family":"Mendez","affiliation":"Centre for Vision Speech and Signal Processing , University of Surrey , Guildford , UK"},{"name":"Chris Russell","slug":"chris-russell","given":"Chris","family":"Russell","affiliation":"Amazon , Tubingen , Germany"},{"name":"Richard Bowden","slug":"richard-bowden","given":"Richard","family":"Bowden","affiliation":"Centre for Vision Speech and Signal Processing , University of Surrey , Guildford , UK"}],"abstract":"We approach instantaneous mapping, converting images to a top-down view of the world, as a translation problem. We show how a novel form of transformer network can be used to map from images and video directly to an overhead map or bird's-eye-view (BEV) of the world, in a single end-to-end network. We assume a 1-1 correspondence between a vertical scanline in the image, and rays passing through the camera location in an overhead map. This lets us formulate map generation from an image as a set of sequence-to-sequence translations. This constrained formulation, based upon a strong physical grounding of the problem, leads to a restricted transformer network that is convolutional in the horizontal direction only. The structure allows us to make efficient use of data when training, and obtains state-of-the-art results for instantaneous mapping of three large-scale datasets, including a 15\\% and 30\\% relative gain against existing best performing methods on the nuScenes and Argoverse datasets, respectively.","doi":"10.24963/ijcai.2023/725","url":"https://ijcai.org/proceedings/2023/725","pdf":"https://ijcai.org/proceedings/2023/0725.pdf","markdown":"https://ijcai.org/proceedings/2023/725.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/725","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":725,"pages":{"first":6486,"last":6491},"keywords":["Sister Conferences Best Papers: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"3rd International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Caesar et al., 2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driv- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11621- 11631, 2020.","title":"nuscenes: A multimodal dataset for autonomous driving","authors":"Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, Oscar Beijbom","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"Proceedings of the European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chang et al., 2019] Ming-Fang Chang, John Lambert, Pat- sorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ra- manan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8748- 8757, 2019.","title":"Argoverse: 3d tracking and forecasting with rich maps","authors":"Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2016] Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monocular 3d object detection for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2147-2156, 2016.","title":"Monocular 3d object detection for autonomous driving","authors":"Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, Raquel Urtasun","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT (1), 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT (1)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In International Conference on Learning Representations, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2021] Anthony Hu, Zak Murez, Nikhil Mo- han, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. FIERY: Future in- stance segmentation in bird's-eye view from surround monocular cameras. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.","title":"FIERY: Future instance segmentation in bird's-eye view from surround monocular cameras","authors":"Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, Alex Kendall","year":"2021","venue":"Proceedings of the International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kehl et al., 2017] Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilic, and Nassir Navab. Ssd-6d: Mak- ing rgb-based 3d detection and 6d pose estimation great again. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 1521-1529, 2017.","title":"Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again","authors":"Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilic, Nassir Navab","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kesten et al., 2019] R Kesten, M Usman, J Houston, T Pandya, K Nadhamuni, A Ferreira, M Yuan, B Low, A Jain, P Ondruska, et al. Lyft level 5 av dataset 2019. urlhttps://level5. lyft. com/dataset, 2019.","title":"Lyft level 5 av dataset","authors":"M R Kesten, J Usman, T Houston, K Pandya, Nadhamuni, M Ferreira, B Yuan, Low, P Jain, Ondruska","year":"2019","venue":null,"doi":null,"url":"urlhttps://level5.lyft.com/dataset","internal_url":null},{"position":12,"raw":"[Lin et al., 2017] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Fea- ture pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.","title":"Feature pyramid networks for object detection","authors":"Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2020] Buyu Liu, Bingbing Zhuang, Samuel Schulter, Pan Ji, and Manmohan Chandraker. Understand- ing road layout from videos as a whole. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 4414-4423, 2020.","title":"Understanding road layout from videos as a whole","authors":"Buyu Liu, Bingbing Zhuang, Samuel Schulter, Pan Ji, Manmohan Chandraker","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lu et al., 2019] Chenyang Lu, Marinus Jacobus Gerardus van de Molengraft, and Gijs Dubbelman. Monocular se- mantic occupancy grid mapping with convolutional varia- tional encoder-decoder networks. IEEE Robotics and Au- tomation Letters, 4(2):445-452, 2019.","title":"Monocular semantic occupancy grid mapping with convolutional variational encoder-decoder networks","authors":"Chenyang Lu, Marinus Jacobus Gerardus Van De Molengraft, Gijs Dubbelman","year":"2019","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mani et al., 2020] Kaustubh Mani, Swapnil Daga, Shub- hika Garg, Sai Shankar Narasimhan, Madhava Krishna, and Krishna Murthy Jatavallabhula. Monolayout: Amodal scene layout from a single image. In The IEEE Win- ter Conference on Applications of Computer Vision, pages 1689-1697, 2020.","title":"Monolayout: Amodal scene layout from a single image","authors":"Kaustubh Mani, Swapnil Daga, Shubhika Garg, Sai Shankar Narasimhan, Madhava Krishna, Krishna Murthy Jatavallabhula","year":"2020","venue":"The IEEE Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mousavian et al., 2017] Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box estimation using deep learning and geometry. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7074-7082, 2017.","title":"3d bounding box estimation using deep learning and geometry","authors":"Arsalan Mousavian, Dragomir Anguelov, John Flynn, Jana Kosecka","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Palazzi et al., 2017] Andrea Palazzi, Guido Borghi, Davide Abati, Simone Calderara, and Rita Cucchiara. Learning to map vehicles into bird's eye view. In International Confer- ence on Image Analysis and Processing, pages 233-243. Springer, 2017.","title":"Learning to map vehicles into bird's eye view","authors":"Andrea Palazzi, Guido Borghi, Davide Abati, Simone Calderara, Rita Cucchiara","year":"2017","venue":"International Conference on Image Analysis and Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pan et al., 2020] Bowen Pan, Jiankai Sun, Ho Yin Tiga Le- ung, Alex Andonian, and Bolei Zhou. Cross-view seman- tic segmentation for sensing surroundings. IEEE Robotics and Automation Letters, 2020.","title":"Cross-view semantic segmentation for sensing surroundings","authors":"Jiankai Bowen Pan, Sun, Ho Yin Tiga, Alex Leung, Bolei Andonian, Zhou","year":"2020","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Philion and Fidler, 2020] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary cam- era rigs by implicitly unprojecting to 3d. In Proceedings of the European Conference on Computer Vision, 2020.","title":"Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d","authors":"Fidler Philion, Jonah Philion, Sanja Fidler","year":"2020","venue":"Proceedings of the European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Poirson et al., 2016] Patrick Poirson, Phil Ammirato, Cheng-Yang Fu, Wei Liu, Jana Kosecka, and Alexander C Berg. Fast single shot detection and pose estimation. In 2016 Fourth International Conference on 3D Vision (3DV), pages 676-684. IEEE, 2016.","title":"Fast single shot detection and pose estimation","authors":"Patrick Poirson, Phil Ammirato, Cheng-Yang Fu, Wei Liu, Jana Kosecka, Alexander C Berg","year":"2016","venue":"Fourth International Conference on 3D Vision (3DV)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Roddick and Cipolla, 2020] Thomas Roddick and Roberto Cipolla. Predicting semantic map representations from images using pyramid occupancy networks. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.","title":"Predicting semantic map representations from images using pyramid occupancy networks","authors":"Cipolla Roddick, Thomas Roddick, Roberto Cipolla","year":"2020","venue":"The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Roddick et al., 2019] Thomas Roddick, Alex Kendall, and Roberto Cipolla. Orthographic feature transform for monocular 3d object detection. In Proceedings of the British Machine Vision Conference (BMVC), 2019.","title":"Orthographic feature transform for monocular 3d object detection","authors":"Thomas Roddick, Alex Kendall, Roberto Cipolla","year":"2019","venue":"Proceedings of the British Machine Vision Conference (BMVC)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Saha et al., 2021] Avishkar Saha, Oscar Mendez, Chris Russell, and Richard Bowden. Enabling spatio-temporal aggregation in birds-eye-view vehicle estimation. In Pro- ceedings of the International Conference on Robotics and Automation, 2021.","title":"Enabling spatio-temporal aggregation in birds-eye-view vehicle estimation","authors":"Avishkar Saha, Oscar Mendez, Chris Russell, Richard Bowden","year":"2021","venue":"Proceedings of the International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulter et al., 2018] Samuel Schulter, Menghua Zhai, Nathan Jacobs, and Manmohan Chandraker. Learning to look around objects for top-view representations of out- door scenes. In Proceedings of the European Conference on Computer Vision (ECCV), pages 787-802, 2018.","title":"Learning to look around objects for top-view representations of outdoor scenes","authors":"Schulter","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sengupta et al., 2012] Sunando Sengupta, Paul Sturgess, L'ubor Ladickỳ, and Philip HS Torr. Automatic dense vi- sual semantic mapping from street-level imagery. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 857-862. IEEE, 2012.","title":"Automatic dense visual semantic mapping from street-level imagery","authors":"Sunando Sengupta, Paul Sturgess, L'ubor Ladickỳ, Philip Hs Torr","year":"2012","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Simonelli et al., 2019] Andrea Simonelli, Samuel Rota Bulo, Lorenzo Porzi, Manuel López-Antequera, and Peter Kontschieder. Disentangling monocular 3d object detec- tion. In Proceedings of the IEEE International Conference on Computer Vision, pages 1991-1999, 2019.","title":"Disentangling monocular 3d object detection","authors":"Andrea Simonelli, Samuel Rota Bulo, Lorenzo Porzi, Manuel López-Antequera, Peter Kontschieder","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2019] Ziyan Wang, Buyu Liu, Samuel Schul- ter, and Manmohan Chandraker. A parametric top-view representation of complex road scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10325-10333, 2019.","title":"A parametric top-view representation of complex road scenes","authors":"Ziyan Wang, Buyu Liu, Samuel Schulter, Manmohan Chandraker","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu et al., 2018] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proceed- ings of the IEEE conference on computer vision and pat- tern recognition, pages 2403-2412, 2018.","title":"Deep layer aggregation","authors":"Fisher Yu, Dequan Wang, Evan Shelhamer, Trevor Darrell","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}