Visual Encoding and Decoding of the Human Brain Based on Shared Features

1 Introduction

In recent years, many studies of the visual information processing mechanism of the human brain, the encoding and decoding of brain signals, and artificial intelligence technology using convolutional neural networks (CNNs) have been mutually enlightening and synergistic [Pei et al., 2019; Rajalingham et al., 2018]. In this context, we built an encoding model for human brain activation based on widely used convolutional neural networks, and then decoded the human brain's visual information by reversing the encoding process. This research can facilitate our understanding of the visual information processing mechanism of the human brain, as well as further explore the relationship between the human brain and deep learning.

* Chao Li and Baolin Liu equally contributed as co-first authors.

† Corresponding Author, email: liubaolin@tsinghua.edu.cn

The framework proposed in this paper has two parts (see Figure 1), the first of which is "encoding." In this part, we assume that only some of the features of the neural network have a clear relationship with human brain signals, which are called shared features. They are obtained by projecting the features of low and high layers of the neural network into their respective shared spaces. Then a non-negative sparse map is established between the shared features and brain activity to achieve the encoding process. The second part of the framework is "decoding," in which visual stimuli are reconstructed through neural network visualization technology, meanwhile, dictionary learning and sparse representation are introduced to improve the robustness of the algorithm. We tested the proposed framework on an open fMRI dataset [Kay et al., 2008, 2011; Naselaris et al., 2009] and the experimental results verified the rationality of the encoding models and demonstrated that this method can reconstruct the basic contours and textures of natural image stimuli.

The main innovations presented in this article are as follows:

•  First, based on known neural cognitive mechanisms and machine learning methods, neural network features are projected onto a shared feature space that can be used to build a better encoding model to predict brain activation. We find that encoding models based on low-level features of a neural network can be more accurate than classic models based on Gabor wavelet features.
•  Second, dictionary learning technology is used to train a corresponding dictionary for the shared features, and the sparse representation method is used to estimate features from measured brain signals. This method is robust to noise from fMRI signals and significantly improves the accuracy of feature estimation.
•  Third, an unconditional model is used to generate an image, and the unbiasedness of the generated model ensures the interpretability of our method. In our proposed method, low-level visual features determine the contours of the generated image, and its principles are easy to explain. High-level features enrich the details of the image, and although their physical meaning is difficult to explain, related effects are constrained by low-level features, thereby making the results controllable.

approximate the observed images [Naselaris et al., 2009] or videos [Nishimoto et al., 2011]. These studies make good use of relevant results of encoding models. However, these methods are difficult to generate images flexibly. Since then, researchers have been trying to achieve image reconstruction based on deep learning. Seeliger et al. first established a mapping between brain activation and hidden variables of a pretrained deep generation network, and then used the deep generation network to generate stimulus images [Seeliger et al., 2018]. Shen et al. first built decoding models to estimate the features of each layer of a deep CNN, and then used a deep generation network to perform iterative calculations for the reconstruction of images [Shen et al., 2019]. The results obtained by their methods have high structural and semantic accuracy, but because of relying on deep generation networks as strong prior information, there may be structural and semantic deviations in the generated results due to prior bias.

This paper is also related to the visualization of neural networks. Here we briefly introduce the feature inversion of convolutional neural networks. Mahendran and Vedaldi proposed a TV-norm-based energy function to reconstruct the natural pre-image from the features of different CNN layers [Mahendran and Vedaldi, 2016]. Since then, image reconstruction methods based on CNN and deep generation networks have been developed to further improve the accuracy of reconstructed images [Dosovitskiy and Brox, 2016a, 2016b]. However, these methods rely on training, which may cause deviations in generated results due to prior bias. Recently, Lempitsky et al. proposed deep image prior, which is an unconditional model and can generate image from a single feature by iterating a full convolutional network [Lempitsky et al., 2018]. Considering this method does not require additional training and can effectively recover images, we use it as a prior to achieve feature-to-image reconstruction.

3 Method

In this study, we extract features from pre-trained AlexNet with Caffe version, then build and test encoding models on a public functional MRI dataset. In this section, we first introduce the functional MRI dataset, and then describe the encoding and decoding methods.

2 Related Work

This work includes the encoding and decoding process, as well as implementing image reconstruction by using the recently developed neural network visualization technology. The related works are described as below.

A good encoding model can describe the information processing mechanism of human visual cortex and can accurately predict brain activities. It has been shown that simple cells in V1 are sensitive to stripes at specific spatial locations, spatial frequencies and orientations [Hubel and Wiesel, 1968; Olshausen and Field, 1996]. Based on this characteristic, Kay et al. used Gabor wavelets to extract the visual features in specific receptive fields, and established a voxel-wise encoding model to predict the activation of early visual areas [Kay et al., 2008]. In addition, some recent studies have found that deep neural networks can also be used to extract effective features for modeling. For example, St-Yves and Naselaris used low-level features extracted from deep neural networks as input to obtain higher encoding accuracy [St-Yves and Naselaris, 2017]. Since then, researchers have extracted hierarchical visual features from deep networks and proved that there is a certain correspondence between the low / high-level features of neural networks and the low-level / high-level visual regions of the human brain [Horikawa and Kamitani, 2017; Wen et al., 2018].

The task of decoding is to recover the information of visual stimulus from the measured brain signals, where reconstructing visual stimuli is a challenging branch. According to the visual stimulus materials, related works can be divided into reconstruction of artificial images and reconstruction of natural images. Artificial images include binary images [Miyawaki et al., 2008] and handwritten characters [Du et al., 2017; van Gerven et al., 2010]. Because this type of images is limited to a specific category, its reconstruction task is relatively simple. However, the reconstruction of natural images does not limit the content of the images, so it requires a higher generalization capability of the reconstruction model. The related research can be traced back to the works of the Gallant Lab team. The idea of these works is to select the images / videos that best match to the measured subject's brain responses from a large data set based on the encoding model to

3.1 Functional MRI Dataset

In this study, we use the fMRI dataset collected by Kay et al. [Kay et al., 2008, 2011; Naselaris et al., 2009]. In the experiment, a 4T scanner was used to obtain fMRI data (BOLD signal) with a spatial resolution of 2 × 2 × 2.5 mm 3 and a TR of 1s. All stimuli are grayscale natural images with circular masks and fixed points. The image size is 500 px × 500 px, and the field of view is 20 × 20° . Two subjects participated. The data set consists of a training set and a test set. The training set contains 1,750 different images and the test set contains 120 new images. In the preprocessing procedure, the response amplitude (single value) of each stimulus image is estimated by deconvolving the response time course of each voxel. For each voxel response in the training set, calculating the ratio between the absolute value of the response and its standard error, and defining the median of the ratio as the voxel's signal-to-noise ratio (SNR) [Kay et al., 2011]. In our study, we further divide the training set into two parts: traincon- which models, encoding the train to used is ଵ݊ݎܶ ing set the evaluate to used is ଶ݊ݎܶ tains 1,575 images; training set models, which contains 175 images. A test set of 120 images is used to evaluate the performance of the decoding method.

3.2 Encoding (Feature to Brain Signals)

Recently, related research has confirmed that there is a certain correspondence between the low/high-level features of convolutional neural networks and the low/high-level visual regions of human brain [Wen et al., 2018]. However, the feature space of the neural network and the representation space of brain activity are unlikely to completely overlap. Therefore, we first give the definition of shared feature space: the intersection between the feature space of a particular layer of a neural network and the representation space of brain activation. Then we try to find the shared features from the features of neural network using a linear transform and use them to estimate brain activation. It can improve the encoding accuracy and can facilitate our understanding of the similarities and differences between human brain and neural network.

Encoding with Low-Level Features

Early literatures supports that early visual areas of the human brain are sensitive to low-level features such as Gabor wavelets [Kay et al., 2013, 2008], and AlexNet's first convolutional layer shares similarity with Gabor wavelets, so we use the features of the first convolution layer as the low-level features to build encoding models for early visual areas. In addition, since the BOLD signal of a voxel reflects the population response characteristics of neurons within the voxel, the response characteristics of a voxel are likely to be related to the features of multiple convolution channels [Li et al., 2018]. In general, we use AlexNet's pool1 feature 𝛷௣௢௢௟ଵ ∈ to try voxel, single a for and feature, initial the as ଶ଻×9଺ ×଻ଶܴ construct an encoding map as follows:

where 𝑥଴ and x are original and down-sampled stimulus, respectively; 𝛷௣௢௢௟ଵሺ𝑥ሻ is the feature of pool1 layer when AlexNet takes 𝑥 as input; ℳ௣௢௢௟ଵ is a linear mapping, which is responsible for projecting the pool1 features into a shared space, and the outputs of it is represented as a vector; <⋅,⋅> is product operation, ࢝ is regression coefficients for the activity brain evoked the is ݎ shared feature, 𝑏 is a constant, when the subject observes the image 𝑥଴ . When ℳ௣௢௢௟ଵ is known, ࢝, 𝑏 can be calculated by Lasso regression.

Here pool1 feature is obtained by convolution and pooling operations with 96 convolution kernels. Therefore, this feature has 96 channels. The convolution kernel of each channel is shown in Figure 2a. Some convolution kernels are probably not significantly related to brain activity, so we use the following steps to evaluate and select the channels:

• and ,ܵ 1) Initialization: Denoting the selected channel set is the feature's tensor form is 𝛷௣௢௢௟ଵ are ݇ and ,݆ ,݅ ௜,௝,௞ , where the indexes of width, height, and channel of feature pool1, ℳ௣௢௢௟ଵሺ𝛷௣௢௢௟ଵ , ͻ͸} {ͳ,ʹ,…, = ܵ respectively. Let ௜,௝,௞ ሻ = ∑ 𝛷௣௢௢௟ଵ ௜,௝,௞ ௞∈𝑆 the Calculate .ܵ of size the is |ܵ| where ,|ܵ/| encoding map for all voxels in early visual areas and let is power predictive whose voxels of number the be ଴ܸ greater than 0.4 (The predictive power of a voxel model is defined as the Pearson correlation between predicted .)ଶ݊ݎܶ and measured voxel responses for the images in ͳ. = 𝑑݊݅ Let
• Taking the number of voxels with high predictive power (predictive power greater than 0.4) as the evaluation index, iteratively calculate the value of this index after removing channel useless most the obtaining thereby ,ܵ a channel in the be ௗ௡௜ܸ let channel, useless most the Remove .ܵ in number of voxels with high predictive power at this time, = 𝑑݊݅ and 𝐶௜௡ௗ be the index of the removed channel. Let +ͳ. 𝑑݊݅
• ͻ͸. = 𝑑݊݅ 3) Repeat 2) until

that seen be can It 2b. Figure in shown is ௗ௡௜ܸ The curve of select we so obtained, is value highest the ͺͷ, = 𝑑݊݅ when ,ܵ the 12 channels remaining at this time as the elements in and let ℳ௣௢௢௟ଵ(𝛷௣௢௢௟ଵ ௜,௝,௞ ) = ∑ 𝛷௣௢௢௟ଵ ௜,௝,௞ ௞∈𝑆 final the build to |ܵ/| encoding models.

Encoding with High-Level Features

High-level features reflect more semantic information, and the encoding and decoding of high-level features can obtain additional information outside early visual areas, so it is very important for reconstruction. According to the research by Wen et al., AlexNet's first fully connected layer (FC6) has relatively high encoding accuracy for V4, MT, PPA, FFA, LO and other brain regions [Wen et al., 2018]. Therefore, we select the FC6 layer as the key layer and extract high-level shared features from it, and then build encoding models based on the features. Due to the lack of knowledge about the activation rules of mid- and high-level visual areas, we use an unsupervised learning method to estimate the shared feature space. Here, we train a non-negative dictionary, and use this dictionary as a projection matrix to map features into the shared space. There are two advantages in this way: Firstly, under the influence of non-negative properties, the elements in the dictionary tend to represent the local structure of features [Lee and Seung, 1999], which makes the dimensions of shared features more independent; Secondly, projection can reduce the dimension of features, which can avoid overfitting in subsequent modeling.

To avoid overfitting, we extract FC6 features from a public image database (Webvision) and train a non-negative dictionary. Denoting the extracted feature sample set is {𝜱𝒇ࢉ𝟔 ࢏ ∈ 𝑫𝒇ࢉ𝟔 dictionary non-negative the and 𝑁}, , . . . ͳ, = ݅ ℝ4଴9଺×ଵ , is obtained by calculating the following formula:

= ݆∀ where 𝒞 ≡ {𝑫 ∈ ℝ௠×௞ : 𝑫 = [ࢊ૚,…, ࢊ࢑], ‖ࢊ࢐ ‖ଶ ൑ ͳ, the is 𝜶࢏ dimension, the reduce to ͹ͲͲ = ݇ set we ,}݇ ͳ,…, representation vector. Denoting ℳ𝑓௖଺ሺ𝜱𝒇ࢉ𝟔ሻ = 𝑫𝒇ࢉ𝟔 𝑻 𝜱𝒇ࢉ𝟔 , then a voxel-wise encoding model can be established by solving the following formula:

term. regularization L2 the is 𝑏ሻ ሺ࢝, ܴ where

3.3 Decoding (Brain Signals to Features to Image)

Estimation of Low-Level Feature

In order to achieve feature estimation, after constructing the encoding map, we first give a measure of encoding loss in the general form as follows:

the are 𝑏௞ and ࢝࢑, ,௞ݎ ,௞݋ where 𝓜 is the shared feature, and predictive power, measured brain activity, mapping coeffiThe respectively. voxel, ht݇ cients, and mapping offset of the shared feature can be estimated just by finding the minimum value of 𝐸, but this method will be affected by the high noise of the fMRI signal and the deviation of the encoding map, making the estimated feature accuracy low. In order to improve the robustness and accuracy of feature estimation, when estimating low-level shared features, we first train a non-negative dictionary for shared feature 𝓜࢖࢕࢕࢒૚, then estimate features using the dictionary. Specifically, we can estimate the low-level shared features by solving the following formula:

constants are ߚ where 𝑫𝓜࢖࢕࢕࢒૚ is the dictionary, 𝜆𝛼 and ߛ used to control the sparsity and variance of 𝜶, respectively. is used to control the variance of 𝓜̂ ࢖࢕࢕࢒૚.

Low-level features are sensitive to the contours and positions of objects in the image (see Section 4.3). When estimating high-level features, we can use the estimated low-level features as a priori information, and simultaneously generate images and estimate high-level features, so that the estimated high-level features and low-level features are coordinated.

High-Level Feature Estimation and Image Reconstruction

After getting low-level features, back-propagation is used to simultaneously reconstruct the stimulus image and estimate high-level features. Without regularization, it is difficult to reconstruct the image. Here we use the deep image prior as the regular term [Lempitsky et al., 2018], the formula is as follows:

where 𝑓𝜃 is a randomly initialized hourglass network with 6 layers, 𝜃 is the parameter, ࢞̂ ∈ ℝ𝑊×𝐻×ଵ is the estimated image, and 𝒛 ∈ ℝ𝑊×𝐻×𝐶 is a random code vector subject to the two the of coefficients the are 𝜎𝑓 and 𝜎௣ ͳ], ,Ͳ[ܷ distribution loss terms, respectively. The above method constructs a nonlinear mapping 𝑓𝜃 by optimizing the parameter 𝜃 in the hourglass network, thereby mapping the random code vector 𝒛 to the image we want to reconstruct. (The code for 𝑓𝜃 is available at https://github.com/Nehemiah-Li/Deep-image-prior.) Although the deep image prior does not have a learning process, the experimental results show that as long as the feature estimation is accurate, back propagation based on the prior can effectively reconstruct the observed image.

4 Experimental Results

4.1 Analysis for Encoding Process

In order to know which brain regions the encoding model can be well applied to, we calculated the correlation coefficient between the predicted responses and measured responses for each voxel model on the test set, and counted the proportion of voxels with significant correlation coefficients (P < 0.01) in each brain region (for comparison, the significance test method is exactly the same as that of literature [Kay et al., 2008]). Figure 3a shows that the proportion is highest in the V1 region, and the more the number of high SNR voxels, the more the number of significant voxels. The distribution of the two subjects shows consistency. In addition, we also compared our models with Gabor-wavelet-based encoding models [Kay et al., 2008] (see Figure 3a). It can be seen that the encoding models based on the low-level shared features have higher accuracy. Because the function of the first convolutional layer is easy to explain, the models based on the lowlevel features have strong interpretability. The proportion of encoding models based on the high-level features in each visual region is shown in Figure 3b. High-level features are used to model all brain regions in the data set (the data set gives a total of 8 visual areas). It can be seen that the high-level features can encode mid- and high-level areas such as V3A, V3B, V4 and lateral occipital area (LOC). A considerable number of encoding models have also been built in undivided (other) visual area (it has a large cardinal number).

4.2 Representation Ability of the Dictionaries

In the process of feature estimation, the dictionary plays an important role. After training the dictionaries, we analyze the key attributes of the dictionaries on another dataset. Table 1 shows the dictionary names, feature dimensions, the root mean square error (RMSE) and correlation coefficients between the normalized actual features and the represented features, the average number of non-zero values in the representation vectors and the atoms in the dictionaries. It can be seen that the dictionary 𝑫𝓜࢖࢕࢕࢒૚ have sufficient representation ability for the low-level shared features and can achieve such representation accuracy using less than 200 non-zero elements. In theory, if the shared space does not completely overlap with the feature space, some information will be lost after the projection transformation. Therefore, as a projection matrix, 𝑫𝒇ࢉ𝟔 has no special requirement for its representation ability.

4.3 Analysis for Decoding Process

Figure 4 shows some reconstructed images in the test set. It is shown that the reconstructed results can capture the contours of the observed images, and the textures in some images are consistent. The reconstruction accuracy of the data of subject 1 is stronger than that of subject 2, which is consistent with the difference between the encoding accuracy of two subjects. Because the estimation accuracy of the shared features directly affects the quality of the reconstructed image, we first analyze the accuracy of feature estimation by comparing the RMSE and correlation between the estimated features and the actual features. Table 2 shows the average values of accuracy indexes of the estimated features in the test set of 120 images. It is shown that the accuracy of estimated features on the data of subject 1 is slightly higher than that on the data of subject 2, and the estimation accuracy of low-level features is higher than that of high-level features.

In addition, in order to qualitatively analyze the impact of low-level features and high-level features on the reconstruction results, we separately use low-level features and highlevel features to reconstruct the image, and the results are shown in Figure 5. We can see that the images reconstructed from low-level features have basic contours, and the positions of edges are more accurate. Moreover, by comparing the reconstruction result with the estimated low-level feature map, we can see that the estimated feature map reflects the main information such as the contour and edge of image. The position of the contours of the images reconstructed by highlevel features alone may not be accurate, but the topological structure is consistent with original image. It may be that the high-level features have a wider receptive field and are more abstract. When the high-level features and low-level features are combined to reconstruct the image, the contour and texture accuracy of the result are significantly improved.

Finally, we compared our method with the method proposed by Seeliger et al. [Seeliger et al., 2018]. To our knowledge, their method is the latest method based on this dataset. For comparison, we use the same quantitative evaluation method for the reconstruction results: performing a behavioral perceptual study using Amazon Mechanical Turk (www.mturk.com). Specifically, in each test, the worker is presented with an original image in the test set, and meanwhile he is required to choose between a real reconstruction and a different reconstruction randomly selected from the test set. This process is repeated ten times for each image in the test set, and different reconstructions randomly selected in each test were used as interference terms. More details about the evaluation method can be seen in [Seeliger et al., 2018]. Table 3 shows the statistical results of this test. Decision accuracy represents the total number of correct decisions in all comparisons. Image accuracy indicates the number of images in the test set that can be correctly identified after a majority vote is applied to ten decisions for each image. If there are at least 8 correct decisions out of 10 comparisons, it means that the image is identifiable. Table 3 shows that our lowest accuracy (S2) is slightly lower than the accuracy achieved by Seeliger et al., while the highest accuracy (S1) and average accuracy are significantly higher than the accuracy of their results. (Note that Seeliger et al. used hyperalignment method to average the multiple subjects' data into a single hyperaligned subject data with improved SNR. In theory, the SNR of the data of subject 1 is closer to that of hyperaligned data.)

5 Conclusion

Based on the concept of shared features, we built encoding models to explore the relationship between the deep CNN and human brain activity, and then proposed a reconstruction method for decoding brain signals. Experimental results show that the low-level shared features of the deep CNN can efficiently encode the early visual areas of the human brain, while the shared features of the fully connected layer can encode the mid- and high-level visual areas of the human brain. The reconstruction method effectively integrates the information of low- and high-level shared features, and nearly half of the reconstructions retain the main features of visual stimuli. Further research on the encoding mechanism of the highlevel visual regions of the human brain and its intrinsic relationship with neural networks will be important for understanding the human brain's visual cognitive mechanism and improving the accuracy of decoding brain information.

Acknowledgments

This work was supported by the National Key Research and Development Program of China (No.2018YFB0204304), National Natural Science Foundation of China (No.U1736219 and No.61571327).