{"type":"ScholarlyArticle","title":"Learning to Act for Perceiving in Partially Unknown Environments","authors":[{"name":"Leonardo Lamanna","slug":"leonardo-lamanna","given":"Leonardo","family":"Lamanna","affiliation":"Fondazione Bruno Kessler , Trento , Italy"},{"name":"Mohamadreza Faridghasemnia","slug":"mohamadreza-faridghasemnia","given":"Mohamadreza","family":"Faridghasemnia","affiliation":"Center for Applied Autonomous Sensor Systems , University of Örebro , Sweden"},{"name":"Alfonso Gerevini","slug":"alfonso-gerevini","given":"Alfonso","family":"Gerevini","affiliation":"Department of Information Engineering , University of Brescia , Italy"},{"name":"Alessandro Saetti","slug":"alessandro-saetti","given":"Alessandro","family":"Saetti","affiliation":"Department of Information Engineering , University of Brescia , Italy"},{"name":"Alessandro Saffiotti","slug":"alessandro-saffiotti","given":"Alessandro","family":"Saffiotti","affiliation":"Center for Applied Autonomous Sensor Systems , University of Örebro , Sweden"},{"name":"Luciano Serafini","slug":"luciano-serafini","given":"Luciano","family":"Serafini","affiliation":"Fondazione Bruno Kessler , Trento , Italy"},{"name":"Paolo Traverso","slug":"paolo-traverso","given":"Paolo","family":"Traverso","affiliation":"Fondazione Bruno Kessler , Trento , Italy"}],"abstract":"Autonomous agents embedded in a physical environment need the ability to correctly perceive the state of the environment from sensory data. In partially observable environments, certain properties can be perceived only in specific situations and from certain viewpoints that can be reached by the agent by planning and executing actions. For instance, to understand whether a cup is full of coffee, an agent, equipped with a camera, needs to turn on the light and look at the cup from the top. When the proper situations to perceive the desired properties are unknown, an agent needs to learn them and plan to get in such situations. In this paper, we devise a general method to solve this problem by evaluating the confidence of a neural network online and by using symbolic planning. We experimentally evaluate the proposed approach on several synthetic datasets, and show the feasibility of our approach in a real-world scenario that involves noisy perceptions and noisy actions on a real robot.","doi":"10.24963/ijcai.2023/609","url":"https://ijcai.org/proceedings/2023/609","pdf":"https://ijcai.org/proceedings/2023/0609.pdf","markdown":"https://ijcai.org/proceedings/2023/609.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/609","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Robotics","track":"Main Track","number":609,"pages":{"first":5485,"last":5493},"keywords":["Planning and Scheduling: PS: Learning in planning and scheduling","Robotics: ROB: Cognitive robotics","Robotics: ROB: Perception"],"videos":[],"references":[{"position":1,"raw":"[Asai and Fukunaga, 2018] Masataro Asai and Alex Fuku- naga. Classical planning in deep latent space: Bridging the subsymbolic-symbolic boundary. In Proceedings of the 32nd AAAI conference on artificial intelligence, vol- ume 32, 2018.","title":"Masataro Asai and Alex Fukunaga. Classical planning in deep latent space: Bridging the subsymbolic-symbolic boundary","authors":"Fukunaga Asai","year":"2018","venue":"Proceedings of the 32nd AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Asai, 2019] Masataro Asai. Unsupervised grounding of plannable first-order logic representation from images. In Proceedings of the 29th International Conference on Au- tomated Planning and Scheduling, volume 29, pages 583- 591, 2019.","title":"Unsupervised grounding of plannable first-order logic representation from images","authors":"Masataro Asai, Asai","year":"2019","venue":"Proceedings of the 29th International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ashari and Ghasemzadeh, 2019] Zhila Esna Ashari and Hassan Ghasemzadeh. Mindful active learning. In Pro- ceedings of the 28th International Joint Conference on Ar- tificial Intelligence, pages 2265-2271, 2019.","title":"Zhila Esna Ashari and Hassan Ghasemzadeh. Mindful active learning","authors":"Ghasemzadeh Ashari","year":"2019","venue":"Proceedings of the 28th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bajcsy et al., 2018] Ruzena Bajcsy, Yiannis Aloimonos, and John K Tsotsos. Revisiting active perception. Au- tonomous Robots, 42(2):177-196, 2018.","title":"Revisiting active perception","authors":"Ruzena Bajcsy, Yiannis Aloimonos, John K Tsotsos","year":"2018","venue":"Autonomous Robots","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Belle et al., 2023] Vaishak Belle, Thomas Bolander, An- dreas Herzig, and Bernhard Nebel. Epistemic planning: Perspectives on the special issue. Artificial Intelligence, 316:103842, 2023.","title":"Epistemic planning: Perspectives on the special issue","authors":"Belle","year":"2023","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bertoli et al., 2006] Piergiorgio Bertoli, Alessandro Cimatti, Marco Roveri, and Paolo Traverso. Strong plan- ning under partial observability. Artificial Intelligence, 170(4-5):337-384, 2006.","title":"Strong planning under partial observability","authors":"Piergiorgio Bertoli, Alessandro Cimatti, Marco Roveri, Paolo Traverso","year":"2006","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bohg et al., 2017] Jeannette Bohg, Karol Hausman, Bharath Sankaran, Oliver Brock, Danica Kragic, Stefan Schaal, and Gaurav Sukhatme. Interactive perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics, 33:1273-1291, 2017.","title":"Interactive perception: Leveraging action in perception and perception in action","authors":"Jeannette Bohg, Karol Hausman, Bharath Sankaran, Oliver Brock, Danica Kragic, Stefan Schaal, Gaurav Sukhatme","year":"2017","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bonet and Geffner, 2011] Blai Bonet and Hector Geffner. Planning under partial observability by classical replan- ning: Theory and experiments. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence, pages 1936-1941, 2011.","title":"Planning under partial observability by classical replanning: Theory and experiments","authors":"Geffner Bonet, Blai Bonet, Hector Geffner","year":"2011","venue":"Proceedings of the 22nd International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2011/324"},{"position":9,"raw":"[Bonet and Geffner, 2014] Blai Bonet and Hector Geffner. Belief tracking for planning with sensing: Width, com- plexity and approximations. Journal of Artificial Intelli- gence Research, 50:923-970, 2014.","title":"Belief tracking for planning with sensing: Width, complexity and approximations","authors":"Geffner ; ] Blai Bonet, Hector Bonet, Geffner","year":"2014","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cakmak et al., 2010] Maya Cakmak, Crystal Chao, and An- drea L Thomaz. Designing interactions for robot active learners. IEEE Transactions on Autonomous Mental De- velopment, 2(2):108-118, 2010.","title":"Maya Cakmak, Crystal Chao, and Andrea L Thomaz. Designing interactions for robot active learners","authors":"Cakmak","year":"2010","venue":"IEEE Transactions on Autonomous Mental Development","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chao et al., 2010] Crystal Chao, Maya Cakmak, and An- drea L Thomaz. Transparent active learning for robots. In 2010 5th ACM/IEEE International Conference on Human- Robot Interaction (HRI), pages 317-324. IEEE, 2010.","title":"Crystal Chao, Maya Cakmak, and Andrea L Thomaz","authors":"Chao","year":"2010","venue":"5th ACM/IEEE International Conference on Human-Robot Interaction (HRI)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dengler et al., 2021] Nils Dengler, Tobias Zaenker, Francesco Verdoja, and Maren Bennewitz. Online object-oriented semantic mapping and map updating. In Proceedings of the 10th European Conference on Mobile Robots, pages 1-7. IEEE, 2021.","title":"Online object-oriented semantic mapping and map updating","authors":"Dengler","year":"2021","venue":"Proceedings of the 10th European Conference on Mobile Robots","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Erol et al., 1994] Kutluhan Erol, James A Hendler, and Dana S Nau. Umcp: A sound and complete procedure for hierarchical task-network planning. In Proceedings of the 2nd International Conference on Artificial Intelligence Planning Systems, pages 249-254, 1994.","title":"Umcp: A sound and complete procedure for hierarchical task-network planning","authors":"Erol","year":"1994","venue":"Proceedings of the 2nd International Conference on Artificial Intelligence Planning Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Grotz, 2021] Markus Grotz. Active Vision for Scene Under- standing. KIT Scientific Publishing, 2021.","title":"Active Vision for Scene Understanding","authors":"Markus Grotz","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hayes and Scassellati, 2014] Bradley Hayes and Brian Scassellati. Discovering task constraints through ob- servation and active learning. In Proceedings of the 27th International Conference on Intelligent Robots and Systems, pages 4442-4449. IEEE, 2014.","title":"Discovering task constraints through observation and active learning","authors":"Bradley Hayes, Brian Scassellati","year":"2014","venue":"Proceedings of the 27th International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Helmert, 2006] Malte Helmert. The fast downward plan- ning system. Journal of Artificial Intelligence Research, 26:191-246, 2006.","title":"The fast downward planning system","authors":"Malte Helmert","year":"2006","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Huang et al., 2010] Sheng-Jun Huang, Rong Jin, and Zhi- Hua Zhou. Active learning by querying informative and representative examples. Proceedings of the 23rd Confer- ence on Neural Information Processing Systems, 23, 2010.","title":"Active learning by querying informative and representative examples","authors":"Sheng-Jun Huang, Rong Jin, Zhi-Hua Zhou","year":"2010","venue":"Proceedings of the 23rd Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Janner et al., 2019] Michael Janner, Sergey Levine, William T. Freeman, Joshua B. Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning about physical interactions with object-centric models. In Proceedings of the 7th International Conference on Learning Representations, 2019.","title":"Reasoning about physical interactions with object-centric models","authors":"Michael Janner, Sergey Levine, William T Freeman, Joshua B Tenenbaum, Chelsea Finn, Jiajun Wu","year":"2019","venue":"Proceedings of the 7th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Konidaris et al., 2018] George Konidaris, Leslie Pack Kael- bling, and Tomás Lozano-Pérez. From skills to sym- bols: Learning symbolic representations for abstract high- level planning. Journal of Artificial Intelligence Research, 61:215-289, 2018.","title":"From skills to symbols: Learning symbolic representations for abstract highlevel planning","authors":"Konidaris","year":"2018","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kulick et al., 2013] Johannes Kulick, Tobias Lang, Marc Toussaint, and Manuel Lopes. Active learning for teach- ing a robot grounded relational symbols. In Proceedings of the 22nd International Joint Conference on Artificial In- telligence, 2013.","title":"Active learning for teaching a robot grounded relational symbols","authors":"Kulick","year":"2013","venue":"Proceedings of the 22nd International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2013/217"},{"position":22,"raw":"[Lamanna et al., 2023] Leonardo Lamanna, Luciano Ser- afini, Mohamadreza Faridghasemnia, Alessandro Saffiotti, Alessandro Saetti, Alfonso Gerevini, and Paolo Traverso. Planning for learning object properties. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.","title":"Planning for learning object properties","authors":"Leonardo Lamanna, Luciano Serafini, Mohamadreza Faridghasemnia, Alessandro Saffiotti, Alessandro Saetti, Alfonso Gerevini, Paolo Traverso","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Migimatsu and Bohg, 2022] Toki Migimatsu and Jeannette Bohg. Grounding predicates through actions. In Proceed- ings of the International Conference on Robotics and Au- tomation, pages 3498--3504. IEEE Press, 2022.","title":"Toki Migimatsu and Jeannette Bohg. Grounding predicates through actions","authors":"Bohg Migimatsu","year":"2022","venue":"Proceedings of the International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Occhipinti et al., 2022] Andrés Occhipinti, Blai Bonet, and Hector Geffner. Learning first-order symbolic planning representations that are grounded. In Proceedings of ICAPS Workshop on Bridging the Gap Between AI Plan- ning and Reinforcement Learning (PRL), 2022.","title":"Learning first-order symbolic planning representations that are grounded","authors":"Andrés Occhipinti, Blai Bonet, Hector Geffner","year":"2022","venue":"Proceedings of ICAPS Workshop on Bridging the Gap Between AI Planning and Reinforcement Learning (PRL)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Petrick and Bacchus, 2002] Ronald P. A. Petrick and Fahiem Bacchus. A knowledge-based approach to planning with incomplete information and sensing. In Proceedings of the 6th International Conference on Artificial Intelligence Planning Systems, pages 212-222. AAAI, 2002.","title":"A knowledge-based approach to planning with incomplete information and sensing","authors":"P A Ronald, Fahiem Petrick, Bacchus","year":"2002","venue":"Proceedings of the 6th International Conference on Artificial Intelligence Planning Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pinto et al., 2016] Lerrel Pinto, Dhiraj Gandhi, Yuanfeng Han, Yong-Lae Park, and Abhinav Gupta. The curious robot: Learning visual representations via physical inter- actions. In Proceedings of the 14th European Conference on Computer Vision, pages 3-18, 2016.","title":"The curious robot: Learning visual representations via physical interactions","authors":"Pinto","year":"2016","venue":"Proceedings of the 14th European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ribes et al., 2015] Arturo Ribes, Jesus Cerquides, Yiannis Demiris, and Ramón López de Mántaras. Active learn- ing of object and body models with time constraints on a humanoid robot. IEEE Transactions on Cognitive and Developmental Systems, 8(1):26-41, 2015.","title":"Arturo Ribes, Jesus Cerquides, Yiannis Demiris, and Ramón López de Mántaras. Active learning of object and body models with time constraints on a humanoid robot","authors":"Ribes","year":"2015","venue":"IEEE Transactions on Cognitive and Developmental Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tarabanis et al., 1995] Konstantinos A Tarabanis, Peter K Allen, and Roger Y Tsai. A survey of sensor planning in computer vision. IEEE transactions on Robotics and Automation, 11(1):86-104, 1995.","title":"A survey of sensor planning in computer vision","authors":"A Konstantinos, Tarabanis, Roger Y Peter K Allen, Tsai","year":"1995","venue":"IEEE transactions on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Taylor et al., 2021] Annalisa T Taylor, Thomas A Berrueta, and Todd D Murphey. Active learning in robotics: A review of control principles. Mechatronics, 77:102576, 2021.","title":"Active learning in robotics: A review of control principles","authors":"Annalisa T Taylor, Thomas A Berrueta, Todd D Murphey","year":"2021","venue":"Mechatronics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ugur and Piater, 2015] Emre Ugur and Justus Piater. Bottom-up learning of object categories, action effects and logical rules: From continuous manipulative ex- ploration to symbolic planning. In Proceedings of the International Conference on Robotics and Automation, pages 2627-2633, 2015.","title":"Bottom-up learning of object categories, action effects and logical rules: From continuous manipulative exploration to symbolic planning","authors":"Emre Ugur, Justus Piater","year":"2015","venue":"Proceedings of the International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zeng et al., 2020] Rui Zeng, Yuhui Wen, Wang Zhao, and Yong-Jin Liu. View planning in robot active vision: A sur- vey of systems, algorithms, and applications. Proceedings of the 6th International Conference on Computational Vi- sual Media, 6:225-245, 2020.","title":"View planning in robot active vision: A survey of systems, algorithms, and applications","authors":"Rui Zeng, Yuhui Wen, Wang Zhao, Yong-Jin Liu","year":"2020","venue":"Proceedings of the 6th International Conference on Computational Visual Media","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}