{"type":"ScholarlyArticle","title":"Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images (Extended Abstract)","authors":[{"name":"Kuniyuki Takahashi","slug":"kuniyuki-takahashi","given":"Kuniyuki","family":"Takahashi","affiliation":"Preferred Networks, Inc."},{"name":"Jethro Tan","slug":"jethro-tan","given":"Jethro","family":"Tan","affiliation":"Preferred Networks, Inc."}],"abstract":"Estimation of tactile properties from vision, such as slipperiness or roughness, is important to effectively interact with the environment. These tactile properties help humans, as well as robots, decide which actions they should choose and how to perform them. We, therefore, propose a model to estimate the degree of tactile properties from visual perception alone (e.g., the level of slipperiness or roughness). Our method extends an encoder-decoder network, in which the latent variables are visual and tactile features. In contrast to previous works, our method does not require manual labeling, but only RGB images and the corresponding tactile sensor data. All our data is collected with a webcam and tactile sensor mounted on the end-effector of a robot, which strokes the material surfaces. We show that our model generalizes to materials not included in the training data.","doi":"10.24963/ijcai.2020/665","url":"https://ijcai.org/proceedings/2020/665","pdf":"https://ijcai.org/proceedings/2020/0665.pdf","markdown":"https://ijcai.org/proceedings/2020/665.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/665","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":665,"pages":{"first":4780,"last":4784},"keywords":["Robotics: Vision and Perception","Robotics: Learning in Robotics"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24902/file_24902.m3u8","length_sec":296,"provider":"hls"}],"references":[{"position":1,"raw":"[Baishya and Bäuml, 2016] Shiv S Baishya and Berthold Bäuml. Robust Material Classification With a Tactile Skin Using Deep Learning. In IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pages 8-15, 2016.","title":"Robust Material Classification With a Tactile Skin Using Deep Learning","authors":"S Shiv, Berthold Baishya, Bäuml","year":"2016","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bell et al., 2015] Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. Material recognition in the wild with the materials in context database. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 3479-3487, 2015.","title":"Material recognition in the wild with the materials in context database","authors":"Sean Bell, Paul Upchurch, Noah Snavely, Kavita Bala","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bergmann-Tiest, 2010] Wouter M. Bergmann-Tiest. Tac- tual Perception of Material Properties. Vision Research, 50(24):2775-2782, 2010.","title":"Tactual Perception of Material Properties","authors":"M Wouter, Bergmann-Tiest","year":"2010","venue":"Vision Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Conneau et al., 2016] Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann Lecun. Very Deep Convolutional Networks for Natural Language Processing. arXiv preprint arXiv:1606.01781, 2016.","title":"Very Deep Convolutional Networks for Natural Language Processing","authors":"Alexis Conneau, Holger Schwenk, Loïc Barrault, Yann Lecun","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dahiya et al., 2013] Ravinder S Dahiya, Philipp Mittendor- fer, Maurizio Valle, Gordon Cheng, and Vladimir J Lumel- sky. Directions Toward Effective Utilization of Tactile Skin: A Review. IEEE Sensors Journal, 13(11):4121- 4138, 2013.","title":"Directions Toward Effective Utilization of Tactile Skin: A Review","authors":"Philipp Ravinder S Dahiya, Maurizio Mittendorfer, Gordon Valle, Vladimir J Cheng, Lumelsky","year":"2013","venue":"IEEE Sensors Journal","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dong et al., 2017] Siyuan Dong, Wenzhen Yuan, and Ed- ward Adelson. Improved GelSight Tactile Sensor for Measuring Geometry and Slip. arXiv preprint arXiv:1708.00922, 2017.","title":"Improved GelSight Tactile Sensor for Measuring Geometry and Slip","authors":"Siyuan Dong, Wenzhen Yuan, Edward Adelson","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fishel and Loeb, 2012] Jeremy A Fishel and Gerald E Loeb. Sensing Tactile Microvibrations with the Bio- Tac-Comparison with Human Sensitivity. In IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), pages 1122- 1127, 2012.","title":"Sensing Tactile Microvibrations with the Bio-Tac-Comparison with Human Sensitivity","authors":"Jeremy A Fishel, Gerald E Loeb","year":"2012","venue":"IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fleming, 2014] Roland W. Fleming. Visual Perception of Materials and Their Properties. Vision Research, 94:62- 75, 2014.","title":"Visual Perception of Materials and Their Properties","authors":"Roland W Fleming","year":"2014","venue":"Vision Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gao et al., 2016] Yang Gao, Lisa Anne Hendricks, Kather- ine J Kuchenbecker, and Trevor Darrell. Deep learning for tactile understanding from visual and haptic data. In 2016 IEEE International Conference on Robotics and Automa- tion (ICRA), pages 536-543. IEEE, 2016.","title":"Deep learning for tactile understanding from visual and haptic data","authors":"Yang Gao, Lisa Anne Hendricks, Katherine J Kuchenbecker, Trevor Darrell","year":"2016","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep Residual Learning for Image Recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Paulino et al., 2017] Tiago Paulino, Pedro Ribeiro, Miguel Neto, Susana Cardoso, Alexander Schmitz, Jose Santos- Victor, Alexandre Bernardino, and Lorenzo Jamone. Low- cost 3-axis Soft Tactile Sensors for the Human-Friendly Robot Vizzy. In IEEE International Conference on Robotics and Automation (ICRA), pages 966-971, 2017.","title":"Lowcost 3-axis Soft Tactile Sensors for the Human-Friendly Robot Vizzy","authors":"Tiago Paulino, Pedro Ribeiro, Miguel Neto, Susana Cardoso, Alexander Schmitz, Jose Santos-Victor, Alexandre Bernardino, Lorenzo Jamone","year":"2017","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Schmitz et al., 2014] Alexander Schmitz, Yusuke Bansho, Kuniaki Noda, Hiroyasu Iwata, Tetsuya Ogata, and Shigeki Sugano. Tactile Object Recognition Using Deep Learning and Dropout. In IEEE-RAS International Con- ference on Humanoid Robots (Humanoids), pages 1044- 1050, 2014.","title":"Tactile Object Recognition Using Deep Learning and Dropout","authors":"Alexander Schmitz, Yusuke Bansho, Kuniaki Noda, Hiroyasu Iwata, Tetsuya Ogata, Shigeki Sugano","year":"2014","venue":"IEEE-RAS International Conference on Humanoid Robots (Humanoids)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Schwartz and Nishino, 2019] Gabriel Schwartz and Ko Nishino. Recognizing material properties from im- ages. IEEE transactions on pattern analysis and machine intelligence, 2019.","title":"Recognizing material properties from images","authors":"Gabriel Schwartz, Ko Nishino","year":"2019","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Takahashi and Tan, 2019] Kuniyuki Takahashi and Jethro Tan. Deep visuo-tactile learning: Estimation of tactile properties from images. In 2019 International Conference on Robotics and Automation (ICRA), pages 8951-8957. IEEE, 2019.","title":"Deep visuo-tactile learning: Estimation of tactile properties from images","authors":"Tan ; ] Kuniyuki Takahashi, Jethro Takahashi, Tan","year":"2019","venue":"2019 International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tanaka and Horiuchi, 2015] Midori Tanaka and Takahiko Horiuchi. Investigating Perceptual Qualities of Static Sur- face Appearance Using Real Materials and Displayed Im- ages. Vision Research, 115:246-258, 2015.","title":"Investigating Perceptual Qualities of Static Surface Appearance Using Real Materials and Displayed Images","authors":"Horiuchi ; ] Midori Tanaka, Takahiko Tanaka, Horiuchi","year":"2015","venue":"Vision Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tomo et al., 2016] Tito Pradhono Tomo, Wai Keat Wong, Alexander Schmitz, Harris Kristanto, Alexandre Sarazin, Lorenzo Jamone, Sophon Somlor, and Shigeki Sugano. A Modular, Distributed, Soft, 3-axis Sensor System for Robot Hands. In IEEE-RAS 16th International Confer- ence on Humanoid Robots (Humanoids), pages 454-460, 2016.","title":"A Modular, Distributed, Soft, 3-axis Sensor System for Robot Hands","authors":"Tito Pradhono Tomo, Wai Keat Wong, Alexander Schmitz, Harris Kristanto, Alexandre Sarazin, Lorenzo Jamone, Sophon Somlor, Shigeki Sugano","year":"2016","venue":"IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Yamaguchi and Atkeson, 2016] Akihiko Yamaguchi and Christopher G Atkeson. Combining Finger Vision and Optical Tactile Sensing: Reducing and Handling Errors While Cutting Vegetables. In IEEE-RAS International Conference on Humanoid Robots (Humanoids), pages 1045-1051, 2016.","title":"Combining Finger Vision and Optical Tactile Sensing: Reducing and Handling Errors While Cutting Vegetables","authors":"Atkeson ; ] Akihiko Yamaguchi, Yamaguchi, Christopher G Atkeson","year":"2016","venue":"IEEE-RAS International Conference on Humanoid Robots (Humanoids)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yanagisawa and Takatsuji, 2015] H. Yanagisawa and K. Takatsuji. Effects of Visual Expectation on Perceived Tactile Perception: An Evaluation Method of Surface Texture with Expectation Effect. International Journal of Design, 9(1), 2015.","title":"Effects of Visual Expectation on Perceived Tactile Perception: An Evaluation Method of Surface Texture with Expectation Effect","authors":"H Yanagisawa, K Takatsuji","year":"2015","venue":"International Journal of Design","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yang et al., 2016] Haolin Yang, Fuchun Sun, Wenbing Huang, Lele Cao, and Bin Fang. Tactile Sequence Based Object Categorization: A Bag of Features Modeled by Linear Dynamic System with Symmetric Transition Ma- trix. In International Joint Conference on Neural Networks (IJCNN), pages 5218-5225, 2016.","title":"Tactile Sequence Based Object Categorization: A Bag of Features Modeled by Linear Dynamic System with Symmetric Transition Matrix","authors":"Yang","year":"2016","venue":"International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yuan et al., 2017a] Wenzhen Yuan, Shaoxiong Wang, Siyuan Dong, and Edward Adelson. Connecting Look and Feel: Associating the Visual and Tactile Properties of Physical Materials. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR17), pages 21-26, 2017.","title":"Connecting Look and Feel: Associating the Visual and Tactile Properties of Physical Materials","authors":"Yuan","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR17)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yuan et al., 2017b] Wenzhen Yuan, Chenzhuo Zhu, Andrew Owens, Mandayam A Srinivasan, and Edward H Adel- son. Shape-independent Hardness Estimation Using Deep Learning and a GelSight Tactile Sensor. In IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 951-958, 2017.","title":"Shape-independent Hardness Estimation Using Deep Learning and a GelSight Tactile Sensor","authors":"Yuan","year":"2017","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}