{"type":"ScholarlyArticle","title":"Decoding EEG by Visual-guided Deep Neural Networks","authors":[{"name":"Zhicheng Jiao","slug":"zhicheng-jiao","given":"Zhicheng","family":"Jiao","affiliation":"Department of Radiology and BRIC , University of North Carolina at Chapel Hill , USA"},{"name":"Haoxuan You","slug":"haoxuan-you","given":"Haoxuan","family":"You","affiliation":"BNRist , KLISS , School of Software , Tsinghua University , China"},{"name":"Fan Yang","slug":"fan-yang","given":"Fan","family":"Yang","affiliation":"Department of Radiology and BRIC , University of North Carolina at Chapel Hill , USA"},{"name":"Xin Li","slug":"xin-li","given":"Xin","family":"Li","affiliation":"Department of Radiology and BRIC , University of North Carolina at Chapel Hill , USA"},{"name":"Han Zhang","slug":"han-zhang","given":"Han","family":"Zhang","affiliation":"Department of Radiology and BRIC , University of North Carolina at Chapel Hill , USA"},{"name":"Dinggang Shen","slug":"dinggang-shen","given":"Dinggang","family":"Shen","affiliation":"Department of Radiology and BRIC , University of North Carolina at Chapel Hill , USA"}],"abstract":"Decoding visual stimuli from brain activities is an interdisciplinary study of neuroscience and computer vision. With the emerging of Human-AI Collaboration, Human-Computer Interaction, and the development of advanced machine learning models, brain decoding based on deep learning attracts more attention. Electroencephalogram (EEG) is a widely used neurophysiology tool. Inspired by the success of deep learning on image representation and neural decoding, we proposed a visual-guided EEG decoding method that contains a decoding stage and a generation stage. In the classification stage, we designed a visual-guided convolutional neural network (CNN) to obtain more discriminative representations from EEG, which are applied to achieve the classification results. In the generation stage, the visual-guided EEG features are input to our improved deep generative model with a visual consistence module to generate corresponding visual stimuli. With the help of our visual-guided strategies, the proposed method outperforms traditional machine learning methods and deep learning models in the EEG decoding task.","doi":"10.24963/ijcai.2019/192","url":"https://ijcai.org/proceedings/2019/192","pdf":"https://ijcai.org/proceedings/2019/0192.pdf","markdown":"https://ijcai.org/proceedings/2019/192.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/192","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Humans and AI","track":"Main track","number":192,"pages":{"first":1387,"last":1393},"keywords":["Humans and AI: Human-Computer Interaction","Humans and AI: Human-AI Collaboration"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learn- ing. In OSDI, volume 16, pages 265-283, 2016.","title":"Tensorflow: A system for large-scale machine learning","authors":"Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard","year":"2016","venue":"OSDI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Adamos et al., 2016] Dimitrios A Adamos, Stavros I Dim- itriadis, and Nikolaos A Laskaris. Towards the bio- personalization of music recommendation systems: A single-sensor eeg biomarker of subjective music prefer- ence. Information Sciences, 343:94-108, 2016.","title":"Towards the biopersonalization of music recommendation systems: A single-sensor eeg biomarker of subjective music preference","authors":"Dimitrios A Adamos, Stavros I Dimitriadis, Nikolaos A Laskaris","year":"2016","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bashivan et al., 2015] Pouya Bashivan, Irina Rish, Mo- hammed Yeasin, and Noel Codella. Learning represen- tations from eeg with deep recurrent-convolutional neural networks. arXiv preprint arXiv:1511.06448, 2015.","title":"Learning representations from eeg with deep recurrent-convolutional neural networks","authors":"Pouya Bashivan, Irina Rish, Mohammed Yeasin, Noel Codella","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cecotti and Graser, 2011] Hubert Cecotti and Axel Graser. Convolutional neural networks for p300 detection with ap- plication to brain-computer interfaces. IEEE transactions on pattern analysis and machine intelligence, 33(3):433- 445, 2011.","title":"Convolutional neural networks for p300 detection with application to brain-computer interfaces","authors":"Graser Cecotti, Hubert Cecotti, Axel Graser","year":"2011","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cowen et al., 2014] Alan S Cowen, Marvin M Chun, and Brice A Kuhl. Neural portraits of perception: reconstruct- ing face images from evoked brain activity. Neuroimage, 94:12-22, 2014.","title":"Neural portraits of perception: reconstructing face images from evoked brain activity","authors":"Marvin M Alan S Cowen, Brice A Chun, Kuhl","year":"2014","venue":"Neuroimage","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255. IEEE, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Du et al., 2017] Changde Du, Changying Du, and Huiguang He. Sharing deep generative representation for perceived image reconstruction from human brain activity. arXiv preprint arXiv:1704.07575, 2017.","title":"Changde Du, Changying Du, and Huiguang He. Sharing deep generative representation for perceived image reconstruction from human brain activity","authors":"Du","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Everingham et al., 2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zis- serman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.","title":"The pascal visual object classes (voc) challenge","authors":"Mark Everingham, Luc Van Gool, K I Christopher, John Williams, Andrew Winn, Zisserman","year":"2010","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Güc ¸lütürk et al., 2017] Yagmur Güc ¸lütürk, Umut Güc ¸lü, Katja Seeliger, Sander Bosch, Rob van Lier, and Mar- cel AJ van Gerven. Reconstructing perceived faces from brain activations with deep adversarial neural decoding. In NIPS, pages 4249-4260, 2017.","title":"Reconstructing perceived faces from brain activations with deep adversarial neural decoding","authors":"Yagmur Güc ¸lütürk, Umut Güc ¸lü, Katja Seeliger, Sander Bosch, Rob Van Lier, Marcel Aj Van Gerven","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Haxby et al., 2001] James V Haxby, M Ida Gobbini, Maura L Furey, Alumit Ishai, Jennifer L Schouten, and Pietro Pietrini. Distributed and overlapping representa- tions of faces and objects in ventral temporal cortex. Sci- ence, 293(5539):2425-2430, 2001.","title":"Distributed and overlapping representations of faces and objects in ventral temporal cortex","authors":"Ida James V Haxby, Maura L Gobbini, Alumit Furey, Jennifer L Ishai, Pietro Schouten, Pietrini","year":"2001","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Haynes and Rees, 2005] John-Dylan Haynes and Geraint Rees. Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nature neuro- science, 8(5):686-691, 2005.","title":"Predicting the orientation of invisible stimuli from activity in human primary visual cortex","authors":"Rees ; John-Dylan Haynes, Geraint Haynes, Rees","year":"2005","venue":"Nature neuroscience","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Horikawa and Kamitani, 2017] Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imag- ined objects using hierarchical visual features. Nature communications, 8:15037, 2017.","title":"Generic decoding of seen and imagined objects using hierarchical visual features","authors":"Kamitani Horikawa, Tomoyasu Horikawa, Yukiyasu Kamitani","year":"2017","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jacobs et al., 2009] Adam L Jacobs, Gene Fridman, Robert M Douglas, Nazia M Alam, Peter E Latham, Glen T Prusky, and Sheila Nirenberg. Ruling out and ruling in neural codes. Proceedings of the National Academy of Sciences, 106(14):5936-5941, 2009.","title":"Ruling out and ruling in neural codes","authors":"L Adam, Gene Jacobs, Robert M Fridman, Douglas, M Nazia, Peter E Alam, Glen T Latham, Sheila Prusky, Nirenberg","year":"2009","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jiao et al., 2018] Zhicheng Jiao, Xinbo Gao, Ying Wang, Jie Li, and Haojun Xu. Deep convolutional neural networks for mental load classification based on eeg data. Pattern Recognition, 76:582-595, 2018.","title":"Deep convolutional neural networks for mental load classification based on eeg data","authors":"Zhicheng Jiao, Xinbo Gao, Ying Wang, Jie Li, Haojun Xu","year":"2018","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 694-711. Springer, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kaneshiro et al., 2015] Blair Kaneshiro, Marcos Perreau Guimaraes, Hyung-Suk Kim, Anthony M Norcia, and Patrick Suppes. A representational similarity analysis of the dynamics of object processing using single-trial eeg classification. PloS one, 10(8):e0135697, 2015.","title":"A representational similarity analysis of the dynamics of object processing using single-trial eeg classification","authors":"Blair Kaneshiro, Marcos Perreau Guimaraes, Hyung-Suk Kim, Anthony M Norcia, Patrick Suppes","year":"2015","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kavasidis et al., 2017] Isaak Kavasidis, Simone Palazzo, Concetto Spampinato, Daniela Giordano, and Mubarak Shah. Brain2image: Converting brain signals into images. In ACM MM, pages 1809-1817. ACM, 2017.","title":"Brain2image: Converting brain signals into images","authors":"Isaak Kavasidis, Simone Palazzo, Concetto Spampinato, Daniela Giordano, Mubarak Shah","year":"2017","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offrey Hinton. Deep learning. Nature, 521(7553):436- 444, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2017] Feng Liu, Shouyi Wang, Jay Rosenberger, Jianzhong Su, and Hanli Liu. A sparse dictionary learning framework to discover discriminative source activations in eeg brain mapping. In AAAI, pages 1431-1437, 2017.","title":"A sparse dictionary learning framework to discover discriminative source activations in eeg brain mapping","authors":"Feng Liu, Shouyi Wang, Jay Rosenberger, Jianzhong Su, Hanli Liu","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In CVPR, pages 3431-3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579-2605, 2008.","title":"Laurens van der Maaten and Geoffrey Hinton","authors":"Hinton Maaten","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Miyawaki et al., 2008] Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, and Hi- roki C Tanabe. Visual image reconstruction from human brain activity using a combination of multiscale local im- age decoders. Neuron, 60(5):915-929, 2008.","title":"Visual image reconstruction from human brain activity using a combination of multiscale local image decoders","authors":"Miyawaki","year":"2008","venue":"Neuron","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Naselaris et al., 2009] Thomas Naselaris, Ryan J Prenger, Kendrick N Kay, Michael Oliver, and Jack L Gallant. Bayesian reconstruction of natural images from human brain activity. Neuron, 63(6):902-915, 2009.","title":"Bayesian reconstruction of natural images from human brain activity","authors":"Thomas Naselaris, Ryan J Prenger, Kendrick N Kay, Michael Oliver, Jack L Gallant","year":"2009","venue":"Neuron","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Nestor et al., 2016] Adrian Nestor, David C Plaut, and Marlene Behrmann. Feature-based face representations and image reconstruction from behavioral and neural data. Proceedings of the National Academy of Sciences, 113(2):416-421, 2016.","title":"Feature-based face representations and image reconstruction from behavioral and neural data","authors":"Adrian Nestor, David C Plaut, Marlene Behrmann","year":"2016","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Palazzo et al., 2017] Simone Palazzo, Concetto Spamp- inato, Isaak Kavasidis, and Daniela Giordano. Genera- tive adversarial networks conditioned by brain signals. In ICCV, pages 3410-3418, 2017.","title":"Generative adversarial networks conditioned by brain signals","authors":"Simone Palazzo, Concetto Spampinato, Isaak Kavasidis, Daniela Giordano","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Park et al., 2011] Jaeyoung Park, Kee-Eung Kim, and Yoon-Kyu Song. A pomdp-based optimal control of p300- based brain-computer interfaces. In AAAI, 2011.","title":"A pomdp-based optimal control of p300based brain-computer interfaces","authors":"Park","year":"2011","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Plis et al., 2014] Sergey M Plis, Devon R Hjelm, Ruslan Salakhutdinov, Elena A Allen, Henry J Bockholt, Jef- frey D Long, Hans J Johnson, Jane S Paulsen, Jessica A Turner, and Vince D Calhoun. Deep learning for neu- roimaging: a validation study. Frontiers in neuroscience, 8, 2014.","title":"Deep learning for neuroimaging: a validation study","authors":"Devon R Sergey M Plis, Ruslan Hjelm, Elena A Salakhutdinov, Henry J Allen, Jeffrey D Bockholt, Hans J Long, Jane S Johnson, Jessica A Paulsen, Vince D Turner, Calhoun","year":"2014","venue":"Frontiers in neuroscience","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Poldrack and Farah, 2015] Russell A Poldrack and Martha J Farah. Progress and challenges in probing the human brain. Nature, 526(7573):371-379, 2015.","title":"Progress and challenges in probing the human brain","authors":"A Russell, Martha J Poldrack, Farah","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Soleymani et al., 2016] Mohammad Soleymani, Sadjad Asghari-Esfeden, Yun Fu, and Maja Pantic. Analysis of eeg signals and facial expressions for continuous emotion detection. IEEE Transactions on Affective Computing, 7(1):17-28, 2016.","title":"Analysis of eeg signals and facial expressions for continuous emotion detection","authors":"Mohammad Soleymani, Sadjad Asghari-Esfeden, Yun Fu, Maja Pantic","year":"2016","venue":"IEEE Transactions on Affective Computing","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Spampinato et al., 2017] Concetto Spampinato, Simone Palazzo, Isaak Kavasidis, Daniela Giordano, Nasim Souly, and Mubarak Shah. Deep learning human mind for automated visual classification. In CVPR, pages 6809-6817, 2017.","title":"Deep learning human mind for automated visual classification","authors":"Concetto Spampinato, Simone Palazzo, Isaak Kavasidis, Daniela Giordano, Nasim Souly, Mubarak Shah","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tzovara et al., 2012] Athina Tzovara, Micah M Murray, Gijs Plomp, and Michael H Herzog. Decoding stimulus-related information from single-trial eeg re- sponses based on voltage topographies. Pattern Recog- nition, 45(6):2109-2122, 2012.","title":"Decoding stimulus-related information from single-trial eeg responses based on voltage topographies","authors":"Athina Tzovara, Micah M Murray, Gijs Plomp, Michael H Herzog","year":"2012","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2019] Xiaodan Zhang, Xinbo Gao, Wen Lu, and Lihuo He. A gated peripheral-foveal convolutional neural network for unified image aesthetic prediction. IEEE Transactions on Multimedia, 2019.","title":"A gated peripheral-foveal convolutional neural network for unified image aesthetic prediction","authors":"Xiaodan Zhang, Xinbo Gao, Wen Lu, Lihuo He","year":"2019","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}