{"type":"ScholarlyArticle","title":"Acoustic NLOS Imaging with Cross Modal Knowledge Distillation","authors":[{"name":"Ui-Hyeon Shin","slug":"ui-hyeon-shin","given":"Ui-Hyeon","family":"Shin","affiliation":"Department of Artificial Intelligence , Sungkyunkwan University , Korea"},{"name":"Seungwoo Jang","slug":"seungwoo-jang","given":"Seungwoo","family":"Jang","affiliation":"Department of Artificial Intelligence , Sungkyunkwan University , Korea"},{"name":"Kwangsu Kim","slug":"kwangsu-kim","given":"Kwangsu","family":"Kim","affiliation":"College of Computing and Informatics , Sungkyunkwan University , Korea"}],"abstract":"Acoustic non-line-of-sight (NLOS) imaging aims to reconstruct hidden scenes by analyzing reflections of acoustic waves. Despite recent developments in the field, existing methods still have limitations such as sensitivity to noise in a physical model and difficulty in reconstructing unseen objects in a deep learning model. To address these limitations, we propose a novel cross-modal knowledge distillation (CMKD) approach for acoustic NLOS imaging. Our method transfers knowledge from a well-trained image network to an audio network, effectively combining the strengths of both modalities. As a result, it is robust to noise and superior in reconstructing unseen objects. Additionally, we evaluate real-world datasets and demonstrate that the proposed method outperforms state-of-the-art methods in acoustic NLOS imaging. The experimental results indicate that CMKD is an effective solution for addressing the limitations of current acoustic NLOS imaging methods. Our code, model, and data are available at https://github.com/shineh96/Acoustic-NLOS-CMKD.","doi":"10.24963/ijcai.2023/156","url":"https://ijcai.org/proceedings/2023/156","pdf":"https://ijcai.org/proceedings/2023/0156.pdf","markdown":"https://ijcai.org/proceedings/2023/156.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/156","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":156,"pages":{"first":1405,"last":1413},"keywords":["Computer Vision: CV: Applications","Machine Learning: ML: Multi-modal learning","Computer Vision: CV: Neural generative models, auto encoders, GANs"],"videos":[],"references":[{"position":1,"raw":"[Albanie et al., 2018] Samuel Albanie, Arsha Nagrani, An- drea Vedaldi, and Andrew Zisserman. Emotion recogni- tion in speech using cross-modal transfer in the wild. In Proceedings of the 26th ACM international conference on Multimedia, pages 292-301, 2018.","title":"Emotion recognition in speech using cross-modal transfer in the wild","authors":"Albanie","year":"2018","venue":"Proceedings of the 26th ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alhashim and Wonka, 2018] Ibraheem Alhashim and Peter Wonka. High quality monocular depth estimation via transfer learning. arXiv preprint arXiv:1812.11941, 2018.","title":"High quality monocular depth estimation via transfer learning","authors":"Ibraheem Alhashim, Peter Wonka","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Aytar et al., 2016] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Advances in neural information process- ing systems, 29, 2016.","title":"Soundnet: Learning sound representations from unlabeled video","authors":"Yusuf Aytar, Carl Vondrick, Antonio Torralba","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Wenzheng Chen, Simon Daneau, Fahim Mannan, and Felix Heide. Steady-state non-line-of-sight imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6790- 6799, 2019.","title":"Steady-state non-line-of-sight imaging","authors":"Chen","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Christensen et al., 2020] Jesper Haahr Christensen, Sascha Hornauer, and X Yu Stella. Batvision: Learning to see 3d spatial layout with two ears. In 2020 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 1581-1587. IEEE, 2020.","title":"Batvision: Learning to see 3d spatial layout with two ears","authors":"Christensen","year":"2020","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gan et al., 2019] Chuang Gan, Hang Zhao, Peihao Chen, David Cox, and Antonio Torralba. Self-supervised moving vehicle tracking with stereo sound. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7053-7062, 2019.","title":"Self-supervised moving vehicle tracking with stereo sound","authors":"Chuang Gan, Hang Zhao, Peihao Chen, David Cox, Antonio Torralba","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2020] Ruohan Gao, Changan Chen, Ziad Al- Halah, Carl Schissler, and Kristen Grauman. Visuale- choes: Spatial image representation learning through echolocation. In European Conference on Computer Vi- sion, pages 658-676. Springer, 2020.","title":"Visualechoes: Spatial image representation learning through echolocation","authors":"Gao","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Geng et al., 2021] Ruixu Geng, Yang Hu, Yan Chen, et al. Recent advances on non-line-of-sight imaging: Conven- tional physical models, deep learning, and new scenes. APSIPA Transactions on Signal and Information Process- ing, 11(1), 2021.","title":"Recent advances on non-line-of-sight imaging: Conventional physical models, deep learning, and new scenes","authors":"Ruixu Geng, Yang Hu, Yan Chen","year":"2021","venue":"APSIPA Transactions on Signal and Information Processing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Grau Chopite et al., 2020] Javier Grau Chopite, Matthias B Hullin, Michael Wand, and Julian Iseringhausen. Deep non-line-of-sight reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 960-969, 2020.","title":"Deep non-line-of-sight reconstruction","authors":"Javier Grau Chopite, Matthias B Hullin, Michael Wand, Julian Iseringhausen","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Heide et al., 2014] Felix Heide, Lei Xiao, Wolfgang Hei- drich, and Matthias B Hullin. Diffuse mirrors: 3d recon- struction from diffuse indirect illumination using inexpen- sive time-of-flight sensors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3222-3229, 2014.","title":"Diffuse mirrors: 3d reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors","authors":"Felix Heide, Lei Xiao, Wolfgang Heidrich, Matthias B Hullin","year":"2014","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Henley et al., 2020] Connor Henley, Tomohiro Maeda, Tristan Swedish, and Ramesh Raskar. Imaging behind oc- cluders using two-bounce light. In European Conference on Computer Vision, pages 573-588. Springer, 2020.","title":"Imaging behind occluders using two-bounce light","authors":"Connor Henley, Tomohiro Maeda, Tristan Swedish, Ramesh Raskar","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jang et al., 2022] Seungwoo Jang, Ui-Hyeon Shin, and Kwangsu Kim. Deep non-line-of-sight imaging using echolocation. Sensors, 22(21):8477, 2022.","title":"Deep non-line-of-sight imaging using echolocation","authors":"Jang","year":"2022","venue":"Sensors","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kadambi et al., 2016] Achuta Kadambi, Hang Zhao, Boxin Shi, and Ramesh Raskar. Occluded imaging with time- of-flight sensors. ACM Transactions on Graphics (ToG), 35(2):1-12, 2016.","title":"Occluded imaging with timeof-flight sensors","authors":"Kadambi","year":"2016","venue":"ACM Transactions on Graphics (ToG)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kirmani et al., 2009] Ahmed Kirmani, Tyler Hutchison, James Davis, and Ramesh Raskar. Looking around the corner using transient imaging. In 2009 IEEE 12th Inter- national Conference on Computer Vision, pages 159-166. IEEE, 2009.","title":"Looking around the corner using transient imaging","authors":"Ahmed Kirmani, Tyler Hutchison, James Davis, Ramesh Raskar","year":"2009","venue":"IEEE 12th International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lindell et al., 2019a] David B Lindell, Gordon Wetzstein, and Vladlen Koltun. Acoustic non-line-of-sight imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6780-6789, 2019.","title":"Acoustic non-line-of-sight imaging","authors":"Lindell","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lindell et al., 2019b] David B Lindell, Gordon Wetzstein, and Matthew O'Toole. Wave-based non-line-of-sight imaging using fast fk migration. ACM Transactions on Graphics (ToG), 38(4):1-13, 2019.","title":"Wave-based non-line-of-sight imaging using fast fk migration","authors":"Lindell","year":"2019","venue":"ACM Transactions on Graphics (ToG)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2020] Xiaochun Liu, Sebastian Bauer, and An- dreas Velten. Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems. Nature com- munications, 11(1):1-13, 2020.","title":"Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems","authors":"Xiaochun Liu, Sebastian Bauer, Andreas Velten","year":"2020","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Maeda et al., 2019] Tomohiro Maeda, Guy Satat, Tristan Swedish, Lagnojita Sinha, and Ramesh Raskar. Re- cent advances in imaging around corners. arXiv preprint arXiv:1910.05613, 2019.","title":"Recent advances in imaging around corners","authors":"Tomohiro Maeda, Guy Satat, Tristan Swedish, Lagnojita Sinha, Ramesh Raskar","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mao et al., 2017] Xudong Mao, Qing Li, Haoran Xie, Ray- mond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceed- ings of the IEEE international conference on computer vi- sion, pages 2794-2802, 2017.","title":"Least squares generative adversarial networks","authors":"Xudong Mao, Qing Li, Haoran Xie, Raymond Yk Lau, Zhen Wang, Stephen Paul Smolley","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[O'Toole et al., 2018] Matthew O'Toole, David B Lindell, and Gordon Wetzstein. Confocal non-line-of-sight imaging based on the light-cone transform. Nature, 555(7696):338-341, 2018.","title":"Confocal non-line-of-sight imaging based on the light-cone transform","authors":"O' Matthew, David B Toole, Gordon Lindell, Wetzstein","year":"2018","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pages 234-241. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"International Conference on Medical image computing and computerassisted intervention","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sarfraz et al., 2021] Fahad Sarfraz, Elahe Arani, and Bahram Zonooz. Knowledge distillation beyond model compression. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 6136-6143. IEEE, 2021.","title":"Knowledge distillation beyond model compression","authors":"Fahad Sarfraz, Elahe Arani, Bahram Zonooz","year":"2020","venue":"25th International Conference on Pattern Recognition (ICPR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shen et al., 2021] Siyuan Shen, Zi Wang, Ping Liu, Zhengqing Pan, Ruiqian Li, Tian Gao, Shiying Li, and Jingyi Yu. Non-line-of-sight imaging via neural transient fields. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 43(7):2257-2268, 2021.","title":"Non-line-of-sight imaging via neural transient fields","authors":"Siyuan Shen, Zi Wang, Ping Liu, Zhengqing Pan, Ruiqian Li, Tian Gao, Shiying Li, Jingyi Yu","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Stanton et al., 2021] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34:6906-6919, 2021.","title":"Does knowledge distillation really work?","authors":"Stanton","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tran et al., 2015] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on com- puter vision, pages 4489-4497, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Valverde et al., 2021] Francisco Rivera Valverde, Juana Va- leria Hurtado, and Abhinav Valada. There is more than meets the eye: Self-supervised multi-object detection and tracking with sound by distilling multimodal knowl- edge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11612- 11621, 2021.","title":"Francisco Rivera Valverde, Juana Valeria Hurtado, and Abhinav Valada. There is more than meets the eye: Self-supervised multi-object detection and tracking with sound by distilling multimodal knowledge","authors":"Valverde","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Velten et al., 2012] Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G Bawendi, and Ramesh Raskar. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Na- ture communications, 3(1):1-8, 2012.","title":"Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging","authors":"Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, G Moungi, Ramesh Bawendi, Raskar","year":"2012","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu et al., 2021] Cheng Wu, Jianjiang Liu, Xin Huang, Zheng-Ping Li, Chao Yu, Jun-Tian Ye, Jun Zhang, Qiang Zhang, Xiankang Dou, Vivek K Goyal, et al. Non-line-of- sight imaging over 1.43 km. Proceedings of the National Academy of Sciences, 118(10):e2024468118, 2021.","title":"Non-line-ofsight imaging over 1.43 km","authors":"Cheng Wu, Jianjiang Liu, Xin Huang, Zheng-Ping Li, Chao Yu, Jun-Tian Ye, Jun Zhang, Qiang Zhang, Xiankang Dou, K Vivek, Goyal","year":"2021","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhao et al., 2018] Mingmin Zhao, Tianhong Li, Moham- mad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, and Dina Katabi. Through-wall human pose es- timation using radio signals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7356-7365, 2018.","title":"Through-wall human pose estimation using radio signals","authors":"Zhao","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhu and Cai, 2022] Dayu Zhu and Wenshan Cai. Fast non- line-of-sight imaging with two-step deep remapping. ACS Photonics, 9(6):2046-2055, 2022.","title":"Fast nonline-of-sight imaging with two-step deep remapping","authors":"Cai ; ] Dayu Zhu, Wenshan Zhu, Cai","year":"2022","venue":"ACS Photonics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}