{"type":"ScholarlyArticle","title":"Active Visual Exploration Based on Attention-Map Entropy","authors":[{"name":"Adam Pardyl","slug":"adam-pardyl","given":"Adam","family":"Pardyl","affiliation":"Jagiellonian University , Faculty of Mathematics and Computer Science; Jagiellonian University , Doctoral School of Exact and Natural Sciences"},{"name":"Grzegorz Rypeść","slug":"grzegorz-rypesc","given":"Grzegorz","family":"Rypeść","affiliation":"Warsaw University of Technology"},{"name":"Grzegorz Kurzejamski","slug":"grzegorz-kurzejamski","given":"Grzegorz","family":"Kurzejamski","affiliation":""},{"name":"Bartosz Zieliński","slug":"bartosz-zielinski","given":"Bartosz","family":"Zieliński","affiliation":""},{"name":"Tomasz Trzciński","slug":"tomasz-trzcinski","given":"Tomasz","family":"Trzcinski","affiliation":""}],"abstract":"Active visual exploration addresses the issue of limited sensor capabilities in real-world scenarios, where successive observations are actively chosen based on the environment. To tackle this problem, we introduce a new technique called Attention-Map Entropy (AME). It leverages the internal uncertainty of the transformer-based model to determine the most informative observations. In contrast to existing solutions, it does not require additional loss components, which simplifies the training. Through experiments, which also mimic retina-like sensors, we show that such simplified training significantly improves the performance of reconstruction, segmentation and classification on publicly available datasets.","doi":"10.24963/ijcai.2023/145","url":"https://ijcai.org/proceedings/2023/145","pdf":"https://ijcai.org/proceedings/2023/0145.pdf","markdown":"https://ijcai.org/proceedings/2023/145.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/145","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":145,"pages":{"first":1303,"last":1311},"keywords":["Robotics: ROB: Robotics and vision","Computer Vision: CV: Machine learning for vision","Machine Learning: ML: Attention models"],"videos":[],"references":[{"position":1,"raw":"[Aloimonos et al., 1988] John Aloimonos, Isaac Weiss, and Amit Bandyopadhyay. Active vision. International jour- nal of computer vision, 1(4):333-356, 1988.","title":"John Aloimonos, Isaac Weiss, and Amit Bandyopadhyay","authors":"Aloimonos","year":"1988","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chai, 2019] Yuning Chai. Patchwork: A patch-wise atten- tion network for efficient object detection and segmenta- tion in video streams. CoRR, abs/1904.01784, 2019.","title":"Patchwork: A patch-wise attention network for efficient object detection and segmentation in video streams","authors":"Yuning Chai, Chai","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Repre- sentations, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Girdhar et al., 2022] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Is- han Misra. Omnivore: A Single Model for Many Visual Modalities. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2022.","title":"Omnivore: A Single Model for Many Visual Modalities","authors":"Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens Van Der Maaten, Armand Joulin, Ishan Misra","year":"2022","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 16000-16009, 2022.","title":"Masked autoencoders are scalable vision learners","authors":"Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jayaraman and Grauman, 2017] Dinesh Jayaraman and Kristen Grauman. Learning to look around. CoRR, abs/1709.00507, 2017.","title":"Learning to look around","authors":"Dinesh Jayaraman, Kristen Grauman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jha et al., 2023] Abhishek Jha, Soroush Seifi, and Tinne Tuytelaars. Simglim: Simplifying glimpse based active visual reconstruction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 269-278, January 2023.","title":"Simglim: Simplifying glimpse based active visual reconstruction","authors":"Abhishek Jha, Soroush Seifi, Tinne Tuytelaars","year":"2023","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2022] Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, and Jiaya Jia. Mat: Mask-aware transformer for large hole image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.","title":"Mat: Mask-aware transformer for large hole image inpainting","authors":"Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, Jiaya Jia","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In European conference on com- puter vision, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Loshchilov and Hutter, 2018] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Inter- national Conference on Learning Representations, 2018.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Naseer et al., 2021] Muzammal Naseer, Kanchana Ranas- inghe, Salman H. Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vi- sion transformers. CoRR, abs/2105.10497, 2021.","title":"Intriguing properties of vision transformers","authors":"Naseer","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pathak et al., 2016] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536-2544, 2016.","title":"Context encoders: Feature learning by inpainting","authors":"Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei A Efros","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ramakrishnan and Grauman, 2018] Santhosh K. Ramakr- ishnan and Kristen Grauman. Sidekick policy learning for active visual exploration. CoRR, abs/1807.11010, 2018.","title":"Sidekick policy learning for active visual exploration","authors":"K Santhosh, Kristen Ramakrishnan, Grauman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ramakrishnan et al., 2021] Santhosh K. Ramakrishnan, Di- nesh Jayaraman, and Kristen Grauman. An exploration of embodied visual exploration. Int. J. Comput. Vis., 129:1616-1649, 2021.","title":"An exploration of embodied visual exploration","authors":"K Santhosh, Dinesh Ramakrishnan, Kristen Jayaraman, Grauman","year":"2021","venue":"Int. J. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rangrej and Clark, 2021] Samrudhdhi B. Rangrej and James J. Clark. Visual attention in imaginative agents. CoRR, abs/2104.00177, 2021.","title":"Visual attention in imaginative agents","authors":"B Samrudhdhi, James J Rangrej, Clark","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rangrej et al., 2022] Samrudhdhi B Rangrej, Chetan L Srinidhi, and James J Clark. Consistency driven sequen- tial transformers attention model for partially observable scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2518- 2527, 2022.","title":"Consistency driven sequential transformers attention model for partially observable scenes","authors":"Chetan L Samrudhdhi B Rangrej, James J Srinidhi, Clark","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sandini and Metta, 2003] Giulio Sandini and Giorgio Metta. Retina-like sensors: motivations, technology and applications. In Sensors and sensing in biology and engineering, pages 251-262. Springer, 2003.","title":"Retina-like sensors: motivations, technology and applications","authors":"Giulio Sandini, Giorgio Metta","year":"2003","venue":"Sensors and sensing in biology and engineering","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Seifi and Tuytelaars, 2019] Soroush Seifi and Tinne Tuyte- laars. Where to look next: Unsupervised active visual ex- ploration on 360° input. CoRR, abs/1909.10304, 2019.","title":"Where to look next: Unsupervised active visual exploration on 360° input","authors":"Tuytelaars Seifi, Soroush Seifi, Tinne Tuytelaars","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Seifi and Tuytelaars, 2020] Soroush Seifi and Tinne Tuyte- laars. Attend and segment: Attention guided active seman- tic segmentation. CoRR, abs/2007.11548, 2020.","title":"Attend and segment: Attention guided active semantic segmentation","authors":"Soroush Seifi, Tinne Tuytelaars","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Seifi et al., 2021] Soroush Seifi, Abhishek Jha, and Tinne Tuytelaars. Glimpse-attend-and-explore: Self-attention for active visual exploration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16137-16146, 2021.","title":"Glimpse-attend-and-explore: Self-attention for active visual exploration","authors":"Soroush Seifi, Abhishek Jha, Tinne Tuytelaars","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Śmieja et al., 2018] Marek Śmieja, Łukasz Struski, Jacek Tabor, Bartosz Zieliński, and Przemysław Spurek. Pro- cessing of missing data by neural networks. Advances in neural information processing systems, 31, 2018.","title":"Processing of missing data by neural networks","authors":"Marek Śmieja, Łukasz Struski, Jacek Tabor, Bartosz Zieliński, Przemysław Spurek","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Song et al., 2015] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567- 576, 2015.","title":"Sun rgb-d: A rgb-d scene understanding benchmark suite","authors":"Song","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wenzel et al., 2021] Patrick Wenzel, Rui Wang, Nan Yang, Qing Cheng, Qadeer Khan, Lukas von Stumberg, Niclas Zeller, and Daniel Cremers. 4seasons: A cross-season dataset for multi-weather slam in autonomous driving. In Zeynep Akata, Andreas Geiger, and Torsten Sattler, ed- itors, Pattern Recognition, pages 404-417, Cham, 2021. Springer International Publishing.","title":"4seasons: A cross-season dataset for multi-weather slam in autonomous driving","authors":"Patrick Wenzel, Rui Wang, Nan Yang, Qing Cheng, Qadeer Khan, Niclas Lukas Von Stumberg, Daniel Zeller, Cremers","year":"2021","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 633-641, 2017.","title":"Scene parsing through ade20k dataset","authors":"Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, Antonio Torralba","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}