{"type":"ScholarlyArticle","title":"Visual Encoding and Decoding of the Human Brain Based on Shared Features","authors":[{"name":"Chao Li","slug":"chao-li","given":"Chao","family":"Li","affiliation":"College of Intelligence and Computing , Tianjin Key Laboratory of Cognitive Computing and Application , Tianjin University , Tianjin , China"},{"name":"Baolin Liu","slug":"baolin-liu","given":"Baolin","family":"Liu","affiliation":"School of Computer and Communication Engineering , University of Science and Technology Beijing , Beijing , China"},{"name":"Jianguo Wei","slug":"jianguo-wei","given":"Jianguo","family":"Wei","affiliation":"College of Intelligence and Computing , Tianjin Key Laboratory of Cognitive Computing and Application , Tianjin University , Tianjin , China"}],"abstract":"Using a convolutional neural network to build visual encoding and decoding models of the human brain is a good starting point for the study on relationship between deep learning and human visual cognitive mechanism. However, related studies have not fully considered their differences. In this paper, we assume that only a portion of neural network features is directly related to human brain signals, which we call shared features. In the encoding process, we extract shared features from the lower and higher layers of the neural network, and then build a non-negative sparse map to predict brain activities. In the decoding process, we use back-propagation to reconstruct visual stimuli, and use dictionary learning and a deep image prior to improve the robustness and accuracy of the algorithm. Experiments on a public fMRI dataset confirm the rationality of the encoding models, and comparing with a recently proposed method, our reconstruction results obtain significantly higher accuracy.","doi":"10.24963/ijcai.2020/103","url":"https://ijcai.org/proceedings/2020/103","pdf":"https://ijcai.org/proceedings/2020/0103.pdf","markdown":"https://ijcai.org/proceedings/2020/103.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/103","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":103,"pages":{"first":738,"last":744},"keywords":["Computer Vision: Biomedical Image Understanding","Humans and AI: Cognitive Modeling","Humans and AI: Human-Computer Interaction","Machine Learning: Deep Learning: Convolutional networks"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25302/file_25302.m3u8","length_sec":301,"provider":"hls"}],"references":[{"position":1,"raw":"[Dosovitskiy and Brox, 2016a] Alexey Dosovitskiy, and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks, In 30th Conference on Neural Information Processing Systems, pages 658-666, 2016.","title":"Generating images with perceptual similarity metrics based on deep networks","authors":"Brox ; Alexey Dosovitskiy, Thomas Dosovitskiy, Brox","year":"2016","venue":"30th Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dosovitskiy and Brox, 2016b] Alexey Dosovitskiy, and Thomas Brox. Inverting Visual Representations with Convolutional Networks, In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4829-4837, June 2016.","title":"Inverting Visual Representations with Convolutional Networks","authors":"Brox ; Alexey Dosovitskiy, Thomas Dosovitskiy, Brox","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Du et al., 2017] Changde Du, Changying Du, and Huiguang He. Sharing deep generative representation for perceived image reconstruction from human brain activity, In Proceedings of the International Joint Conference on Neural Networks, pages 1049-1056, May 2017.","title":"Changde Du, Changying Du, and Huiguang He. Sharing deep generative representation for perceived image reconstruction from human brain activity","authors":"Du","year":"2017","venue":"Proceedings of the International Joint Conference on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Horikawa and Kamitani, 2017] Tomoyasu Horikawa, and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, 8: 1-15, 2017.","title":"Generic decoding of seen and imagined objects using hierarchical visual features","authors":"Kamitani ; Tomoyasu Horikawa, Yukiyasu Horikawa, Kamitani","year":"2017","venue":"Nature Communications","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hubel and Wiesel, 1968] D. H. Hubel, and T N Wiesel. Receptive Fields and Functional Architecture of monkey striate cortex. Journal of Physiology, 195: 215-243, 1968.","title":"Receptive Fields and Functional Architecture of monkey striate cortex","authors":"D H Hubel, T N Wiesel","year":"1968","venue":"Journal of Physiology","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kay et al., 2008] Kendrick N. Kay, Thomas Naselaris, Ryan J. Prenger, and Jack L. Gallant. Identifying natural images from human brain activity. Nature, 452: 352-355, March 2008.","title":"Identifying natural images from human brain activity","authors":"Kendrick N Kay, Thomas Naselaris, Ryan J Prenger, Jack L Gallant","year":"2008","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kay et al., 2013] Kendrick N. Kay, Jonathan Winawer, Ariel Rokem, Aviv Mezer, and Brian A. Wandell. A Two-Stage Cascade Model of BOLD Responses in Human Visual Cortex. PLoS Computational Biology, 9: e1003079, May 2013.","title":"A Two-Stage Cascade Model of BOLD Responses in Human Visual Cortex","authors":"Kendrick N Kay, Jonathan Winawer, Ariel Rokem, Aviv Mezer, Brian A Wandell","year":"2013","venue":"PLoS Computational Biology","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kay et al., 2011] Kendrick N Kay, Thomas Naselaris, and Jack L Gallant. fMRI of human visual areas in response to natural images. CRCNS.org, 2011.","title":"fMRI of human visual areas in response to natural images","authors":"Thomas Kendrick N Kay, Jack L Naselaris, Gallant","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lee and Seung, 1999] Daniel D. Lee, and H. Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401: 788-791, 1999.","title":"Learning the parts of objects by non-negative matrix factorization","authors":"D Daniel, H Sebastian Lee, Seung","year":"1999","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lempitsky et al., 2018] Victor Lempitsky, Andrea Vedaldi, and Dmitry Ulyanov. Deep Image Prior, In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9446-9454, June 2018.","title":"Deep Image Prior","authors":"Victor Lempitsky, Andrea Vedaldi, Dmitry Ulyanov","year":"2018","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2018] Chao Li, Junhai Xu, and Baolin Liu. Decoding natural images from evoked brain activities using encoding models with invertible mapping. Neural Networks, 105: 227-235, September 2018.","title":"Decoding natural images from evoked brain activities using encoding models with invertible mapping","authors":"Chao Li, Junhai Xu, Baolin Liu","year":"2018","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mahendran and Vedaldi, 2016] Aravindh Mahendran, and Andrea Vedaldi. Visualizing Deep Convolutional Neural Networks Using Natural Pre-images. International Journal of Computer Vision, 120: 233-255, December 2016.","title":"Visualizing Deep Convolutional Neural Networks Using Natural Pre-images","authors":"Vedaldi ; Aravindh Mahendran, Andrea Mahendran, Vedaldi","year":"2016","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Miyawaki et al., 2008] Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki C Tanabe, Norihiro Sadato, and Yukiyasu Kamitani. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron, 60: 915-929, 2008.","title":"Visual image reconstruction from human brain activity using a combination of multiscale local image decoders","authors":"Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-Aki Sato, Yusuke Morito, Norihiro Hiroki C Tanabe, Yukiyasu Sadato, Kamitani","year":"2008","venue":"Neuron","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Naselaris et al., 2009] Thomas Naselaris, Ryan J. Prenger, Kendrick N. Kay, Michael Oliver, and Jack L. Gallant. Bayesian Reconstruction of Natural Images from Human Brain Activity. Neuron, 63: 902-915, 2009.","title":"Bayesian Reconstruction of Natural Images from Human Brain Activity","authors":"Thomas Naselaris, Ryan J Prenger, Kendrick N Kay, Michael Oliver, Jack L Gallant","year":"2009","venue":"Neuron","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nishimoto et al., 2011] Shinji Nishimoto, An T. Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L. Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology, 21: 1641- 1646, 2011.","title":"Reconstructing visual experiences from brain activity evoked by natural movies","authors":"Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, Jack L Gallant","year":"2011","venue":"Current Biology","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Olshausen and Field, 1996] Bruno A Olshausen, and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381: 607, 1996.","title":"Emergence of simple-cell receptive field properties by learning a sparse code for natural images","authors":"David J Bruno A Olshausen, Field","year":"1996","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pei et al., 2019] Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, Feng Chen, Ning Deng, Si Wu, Yu Wang, Yujie Wu, Zheyu Yang, Cheng Ma, Guoqi Li, Wentao Han, Huanglong Li, Huaqiang Wu, Rong Zhao, Yuan Xie, and Luping Shi. Towards artificial general intelligence with hybrid Tianjic chip architecture. Nature, 572: 106-111, August 2019.","title":"Towards artificial general intelligence with hybrid Tianjic chip architecture","authors":"Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, Feng Chen, Ning Deng, Si Wu, Yu Wang, Yujie Wu, Zheyu Yang, Cheng Ma, Guoqi Li, Wentao Han, Huanglong Li, Huaqiang Wu, Rong Zhao, Yuan Xie, Luping Shi","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rajalingham et al., 2018] Rishi Rajalingham, Elias B. Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J. DiCarlo. Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks. The Journal of Neuroscience, 38: 7255-7269, August 2018.","title":"Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks","authors":"Rishi Rajalingham, Elias B Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, James J Dicarlo","year":"2018","venue":"The Journal of Neuroscience","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Seeliger et al., 2018] Katja Seeliger, Umut Güçlü, Luca Ambrogioni, Yagmur Güçlütürk, and M.A.J. van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage, 181: 775-785, November 2018.","title":"Generative adversarial networks for reconstructing natural images from brain activity","authors":"Katja Seeliger, Umut Güçlü, Luca Ambrogioni, Yagmur Güçlütürk, M A J Van Gerven","year":"2018","venue":"NeuroImage","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shen et al., 2019] Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image reconstruction from human brain activity. PLoS Computational Biology, 15: e1006633, January 2019.","title":"Deep image reconstruction from human brain activity","authors":"Guohua Shen, Tomoyasu Horikawa, Kei Majima, Yukiyasu Kamitani","year":"2019","venue":"PLoS Computational Biology","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[St-Yves and Naselaris, 2017] Ghislain St-Yves, and Thomas Naselaris. The feature-weighted receptive field: An interpretable encoding model for complex feature spaces. NeuroImage, 2017.","title":"The feature-weighted receptive field: An interpretable encoding model for complex feature spaces","authors":"Ghislain St-Yves, Thomas Naselaris","year":"2017","venue":"NeuroImage","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[van Gerven et al., 2010] Marcel A.J. van Gerven, Floris P. de Lange, and Tom Heskes. Neural decoding with hierarchical generative models. Neural Computation, 22: 3127-3142, December 2010.","title":"Neural decoding with hierarchical generative models","authors":"A J Marcel, Floris P Van Gerven, Tom De Lange, Heskes","year":"2010","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wen et al., 2018] Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision. Cerebral Cortex, 28: 4136- 4160, December 2018.","title":"Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision","authors":"Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, Zhongming Liu","year":"2018","venue":"Cerebral Cortex","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}