Learning Regional Attention Convolutional Neural Network for Motion Intention Recognition Based on EEG Data

1 Introduction

The Brain-Computer Interface (BCI) can be defined as a technology that translates brain signals into commands for interactive applications [Cecotti and Graser, 2010]. Noninvasive Electroencephalography (EEG) is regarded as one of the most convenient ways to record brain activity. EEG-based motion intention recognition algorithms promise to revolutionize many application areas [Shan et al., 2018], notably to enable severely motor-impaired users to control assistive technologies [Schiatti et al., 2018] (e.g., mind-controlled exoskeletons, rehabilitation robotics, and entertainments). Due to low signal-to-noise ratio, limited public EEG datasets, it's very challenging to tackle this task.

∗ corresponding author.

In the past few years, many EEG-based motion intention recognition algorithms have been proposed, which mainly focus on two parts: handcrafted feature-based models and deep learning-based models. Features extracted by common spatial pattern (CSP) algorithm [Wu et al., 2014] or discrete wavelet transform (DWT) [Chen et al., 2016] are fed to classical machine learning models such as support vector machine (SVM) [Lal et al., 2005], linear discriminant analysis (LDA) [Kaneshiro et al., 2015]. Deep learning-based models use deep neural networks to learn discriminative and robust features in an end-to-end manner [Sakhavi et al., 2018]. For example, [Zhang et al., 2018] try to learn spatiotemporal features by using cascade and parallel convolutional recurrent neural networks. [Chen et al., 2018] extract EEG signals with different frequencies and introduce a novel multi-task deep learning model to learn human intentions. However, most of them fail to take full advantage of spectral-spatial-temporal features, which are all essential cues for EEG-based motion intention recognition [Bashivan et al., 2016].

In this paper, we effectively and explicitly use the spectralspatial-temporal features and propose a novel regional attention convolutional neural network. Firstly, we use timefrequency-based approaches to analyze EEG data as a multidimensional signal that contains frequency as a prominent dimension, which provides many opportunities to link EEG data to experimental manipulations and ongoing subject behaviour. By doing so, we can explore spectral features of different brain regions while decreasing temporal precision to a certain extent. According to neuroscience, when a person performs motor execution (ME) or motor imagery (MI), the amplitude or energy of mu and beta rhythms in specific area of the primary sensorimotor will decrease, resulting in event-related desynchronization (ERD) [Dornhege et al., 2003]. For example, when a person executes both feet, righthand or left-hand motor movement, the energy of mu and beta rhythms in Cz, C3 or C4 decrease, separately. As shown in Figure 1, many task-relevant dynamics in EEG data are revealed by time-frequency-based approaches. Secondly, we propose a novel regional attention convolutional neural network (RACNN) to capture the importance of brain regions for EEG-based motion intention recognition. The RACNN consists of feature extraction module, self-attention module, and regional attention module. Given EEG data of different brain region, RACNN learns attention weights for each area in an end-to-end manner, and aggregates their CNN-based features. Finally, since functional asymmetry [Grosse-wentrup, 2009] of the cerebral hemisphere, we propose an attention loss to encourage a high attention weight for the most critical brain region. The attention loss constraint on the RACNN that the attention of the activated brain region should be larger than others.

Our main contributions can be summarized as below:

1. We propose a novel regional attention convolutional neural network (RACNN) to explore spectral-spatialtemporal features for EEG motion intention recognition. RACNN aggregates a varied number of spatial features of different brain regions produced by a backbone convolutional neural network into a compact fixedlength representation. The learned features are sensitive to task-relevant dynamics by distinguishing spectraltemporal features of different brain regions.
2. We adopt time-frequency based analysis to reveal spectral-temporal features in terms of neural oscillations of primary sensorimotor. It can stably reveal many taskrelevant dynamics in EEG data of different brain regions.
3. Experiments on three datasets: PhysioNet EEG dataset, Upper Limb Movement EEG dataset and real-world BCI dataset demonstrate the effectiveness of our proposed RACNN and time-frequency based analysis. Based on them, our approach outperforms previous methods.

2 RelatedWorks

2.1 Handcrafted Feature-BasedModels

Previous EEG analysis systems learn hand-crafted features. [Chen et al., 2016] use discrete wavelet transform (DWT) to extract features for epileptic focus localization. An optimal time-frequency resolution can be achieved in all frequency ranges by adopting DWT because of its varying window size. [Park and Chung, 2019] propose a novel motion intention recognition algorithm using filter-bank common spatial pattern (FBCSP) features. Those methods prove the effectiveness of spectral-spatial-temporal features for analyzing EEG signals. However, those features are manually designed, which is time-consuming and highly relies on the human experience. Different from them, our model exploits spectralspatial-temporal information with deep learning model.

2.2 Deep Learning-BasedModels

Deep Learning methods have been rising in popularity in the past few years, and are used as the most effective machine learning technology in dealing with EEG signals. Some deep learning-based models learn spatial-temporal representations of raw EEG data. For example, cascade and parallel convolutional recurrent network models [Zhang et al., 2018] are used to extract spatial-temporal information and generate robust representations. [Chen et al., 2018] extract EEG signals of different frequency band and introduce a novel Multitask deep learning model to learn human intentions. Apart from those spatial-temporal features decoding method, the proposed approach [Sakhavi et al., 2018] is used to preserve part spectral feature and spatial-temporal features of EEG data which leads to finding features that are less sensitive to variations within each dimension. Different from those deep learning-based models, which mainly focus on spatialtemporal features and fail to take advantage of spectral information, our regional attention convolutional neural network explicitly explores the spectral-spatial-temporal features.

2.3 Time-Frequency Based Analysis

Time-frequency-based approaches are used to conceptualize and analyze EEG data as a multidimensional signal that contains frequency as a prominent dimension, which provides many opportunities to link EEG data to experimental manipulations and ongoing subject behaviour. Recently, by using time-frequency-based approaches to reveal neural oscillations of primary sensorimotor, some methods [Tang et al., 2016] have shown remarkable results in BCI. Different from them, our time-frequency-based approach is specifically designed for motion intention recognition, which adopts complex Morlet wavelets and baseline normalization, thus generating stable time-frequency representations.

3 Regional Attention Convolutional Neural Network

3.1 Generating Time-Frequency Representation

To generate time-resolved frequency representation of EEG data, we adopt band-pass filter (4-30 Hz), complex Morlet wavelet convolution, and baseline normalization. With complex Morlet wavelet convolution, the mapping between the two vectors is represented in a 2-D space that allows extracting not only the bandpass filtered signal but also timefrequency power and phase information [Cohen, 2014].

A complex Morlet wavelet ω can be defined as the product of a complex sine wave and a Gaussian window.

where i is the imaginary operator, f is frequency, and t is time. To avoid introducing a phase shift, t should be centred at t = 0, σ is the width of the Gaussian, which is defined as σ = n 2πf . When computing the dot product between a k Hz complex wavelet ω and a single-trial EEG data ϕ(t) at time s, the result of the dot product contains a real part and an imaginary part.

By extracting the magnitude of the result of convolution, we construct a time series of power one frequency band. Taking frequency into account, we can obtain time-resolved frequency representations of EEG data. It's still difficult to visualize activity from a large range of frequency bands simultaneously based on raw power values. Baseline normalization is used to convert time-resolved frequency representations to a scale that is suitable for quantitative statistical analysis. Decibel normalization and baseline subtraction are applied separately.

where atf is frequency-specific power, and btf is the baseline level of power in that same frequency. The generated timeresolved frequency representations are shown in Figure 1.

3.2 Regional Attention Convolutional Neural Network

After using complex Morlet wavelets and baseline normalization to extract the time-varying power from EEG data of each channel, the time-resolved frequency representations R are created as follows:R = [r1, r2 . . . rn] (5)

where rn is the time-frequency representation of channel n, and n is the number of channels. Regional attention convolutional neural network (RACNN) is used to recognize motion intentions from each time-resolved frequency representations R. RACNN consists of feature extraction module, self-attention module, and regional attention module as illustrated in Figure 3. The input to the model is the preprocessed representations of 3D data (e.g., R), containing both spectralspatial-temporal information. We first extract the spectraltemporal features of each data representation and coarsely calculate the importance of each channel by a global average pooling layer conducted on its feature, which is called self-attention module. The second stage seeks to find more accurate attention weights by modeling the relationship between varied representations of different brain regions aggregated from the first stage, which is called regional attention module.

To extract the spectral-temporal features of each data, we apply a backbone CNN, as shown in Figure 3. The input time-resolved frequency representations are defined as R = [r1, r2 . . . rn]. The feature representation rn extracted by CNN is defined as below:

Through the CNN spectral-temporal feature extraction step, the input time-resolved frequency representations are transformed into sequences of spectral-temporal feature representations.

Self-attention module. To coarsely calculate the importance of each channel, we apply a global average pooling layer with a tanh activation. Mathematically, the attention weight of the n-th channel is defined as below:

where g denotes the global average pooling function, and f denotes the tanh activation function. In this stage, we aggregate the spectral-temporal features of the same brain region with their attention weights into a global representation.

where F j r is the spatial feature of the j-th brain region, which contains k channels.

Regional attention module. Since the aggregated representation F j r , j = 1, 2 . . .m inherently represents the contents of all brain regions, the attention weights can be further refined by modeling the relationship between varied representations of different brain regions. With these region features, the region attention module applies an FC layer and a softmax function to estimate the attention weights βj of j-th brain region.

where f denotes the softmax function, and q1 is the parameter of fully connected layer. The attention information along with the spectral-temporal features can be aggregated into a new compact feature.

where λi is the minimum self-attention of i-th brain region, λi = min (αk . . . αd). The spectral-spatial-temporal representation F is fed into a standard softmax classifier. Then the classification loss can be formulated as:

Regional attention loss. Inspired by the neuroscience findings that functional asymmetry of the cerebral hemisphere, we propose a region biased loss to encourage high attention weights for the most important regions. The region attention loss can be formulated as:

where δ is hyper-parameter served as margin, αmax and αmin is the maximum and minimum value of self-attention.

Finally, the overall loss function of our regional attention convolutional neural network can be formulated as:

4 Experiment

4.1 Datasets

PhysioNet EEG dataset [Goldberger et al., 2000] contains 109 subjects with 64 electrode channels and 160Hz [Schalk et al., 2004] sampling rate. Each subject performed motor execution (ME) and motor imagery (MI) to finish four-class intentions, including eye closed, both feet, right fist and left fist open and close. Nevertheless, we found that the data of the #88, #89, #92, #100, and #104 subject were damaged in the data processing stage, so this participant's record was removed.

Upper Limb Movement EEG dataset [Ofner et al., 2017] contains 15 subjects with 61 electrode channels and 512 Hz sampling rate. Each subject performed motor execution (ME) and motor imagery (MI) to finish seven-class intentions, including rest, elbow flexion, elbow extension, forearm supination, forearm pronation, hand close, and hand open.

4.2 Implementation Details

Nine electrode channels, which are shown in Figure 4, are chosen to generate time-resolved frequency representations. Through band-pass filter (4-30 Hz), complex Morlet wavelet convolution, and baseline normalization, the raw 4 trials of EEG data from the same session are used to generate spectralspatial-temporal representation, as shown in Figure 1. Then three-quarters generated data chosen in random are used as the training set, and others are used as the validation set.

For CNN part, we adopt three convolution layers with kernel sizes of 3×3 to extract spectral-temporal features. For the classification loss, a classifier includes a 1024-dim FC layer as the middle layer followed by dropout and an FC layer with identity number logits as output layer. The keep probability of the dropout operation is 0.6.

All experiments are established in TensorFlow framework with batch size 100. When jointly training with regional attention loss and classification loss, the default loss weight ratio is 2:1. The margin in regional attention loss is default as 0.15. The Adam method with 10-5 learning rate is used as the optimizer.

4.3 Ablation Study

To evaluate each component of our regional attention convolutional neural network, we conduct two variants trained under different settings. Firstly, we train the model under Eq.(12) as a baseline. Secondly, by comparing the whole model under Eq.(14) with the first variants, we can verify the effectiveness of attention loss. Both the first two variants are learning from time-frequency representations generated by complex Morlet wavelets and baseline subtraction under Eq.(4). Thirdly, by comparing model learning from time-frequency representations produced by complex Morlet wavelets and decibel normalization under Eq.(3) with the second variants, we can verify the effectiveness of complex Morlet wavelets and baseline subtraction. We mark them as Lcls(sub), Lall(sub), and Lall(db), respectively.

Comparison with different variants of our regional attention convolutional neural network on PhysioNet EEG dataset and Upper Limb Movement EEG dataset as shown in Table 1, Lcls(sub) achieves 74.6% and 39.2% classification accuracy of ME and 68.3% and 30.2% classification accuracy of MI on PhysioNet EEG dataset and Upper Limb Movement EEG dataset, respectively. This demonstrates that the regional attention convolutional neural network is capable of learning spectral-spatial-temporal features generated by time-frequency-based approaches. Secondly, Lall(sub) achieves 76.9% and 42.6% classification accuracy of ME and 70.2% and 33.1% classification accuracy of MI on PhysioNet EEG dataset and Upper Limb Movement EEG dataset, respectively. Compared with Lcls(sub), the attention loss improves the result by 2.3% and 3.4% classification accuracy of ME and 1.9% and 2.9% classification accuracy of MI on PhysioNet EEG dataset and Upper Limb Movement EEG dataset, respectively. This shows that attention loss contributes to learning spectral-spatial-temporal features. Finally, Lall(db) achieves 70.1% and 37.9% classification accuracy of ME and 65.8% and 27.4% classification accuracy of MI on PhysioNet EEG dataset and Upper Limb Movement EEG dataset, respectively. Compared with Lall(db), complex Morlet wavelets and baseline subtraction improves the result by 6.8% and 4.7% classification accuracy of ME and 4.4% and 5.7% classification accuracy of MI on PhysioNet EEG dataset and Upper Limb Movement EEG dataset, respectively. This shows that complex Morlet wavelets and baseline subtraction contribute to learning spectral-spatialtemporal features. In summary, our proposed approach by using spectral-spatial-temporal information is effective for motion intention recognition.

4.4 Evaluation of Time-Resolved Frequency Representation

Figure 1 displays the time-frequency representations generated by our complex Morlet wavelet convolution and baseline subtraction for ME of three movements from subject 3.

For each movement, time-frequency representations of channel C4 over the right sensorimotor cortex, C3 over the left sensorimotor cortex and Cz are demonstrated. In each time-frequency representation, the cue occurs at 0 s. The blue color means ERD (power decrease), and the yellow color means ERS (power increase). For both feet movement, the ERD in both theta (4-7 Hz) and alpha (8-13 Hz) bands is shown at Cz electrode and the ERS in both alpha and beta bands is shown at the hand area (C3 and C4). For the right fist and left fist movement, ERD is shown from around 0-4 s after the cue onset due to the response delay; ERD in both theta and alpha bands is shown over motor areas. What's more, ERD is most apparent over motor areas contralateral to the hand moves. In summary, our complex Morlet wavelet convolution can stably reveal many task-relevant dynamics in EEG data of different brain regions.

4.5 Visualizing AttentionWeights

We present the visualized analysis of recognizing both feet, right-hand or left-hand motor execution on PhysioNet EEG dataset. Fig.4(a) shows the positions of different electrodes for PhysioNet EEG dataset; Fig.4(b) shows the attention weights of different electrodes for both feet, left-fist or rightfist motor execution (ME).

Through self-attention module and regional attention module, our proposed regional attention convolutional neural network puts a high emphasis on Cz, FC4, and C3 when recognizing both feet, right-hand or left-hand motor execution, respectively. Compared with simple aggregation mechanism, which treats all time-frequency representations equally, the proposed attention mechanism automatically learns the priority of different brain region, which works better in feature fusion, and achieves better performance.

4.6 Comparison with Previous Methods

We compare our regional attention convolutional neural network with state-of-the-art, which can be grouped into two groups, i.e. handcrafted feature-based models, and deep learning-based models. The compared methods are used to perform single-trial classification.

The experimental results on PhysioNet EEG dataset and Upper Limb Movement EEG dataset are shown in Table 2 and Table 3. Firstly, the handcrafted feature-based models contribute to motion intention recognition but achieve a poor classification performance. This is mainly because handcrafted feature-based models can't take full advantage of big data. Secondly, deep learning-based models significantly outperform the handcrafted feature-based models, which shows the effectiveness of deep learning. Finally, our regional attention convolutional neural network outperforms all those methods achieving new state-of-the-art. The experimental results verify the superiority of regional attention convolutional neural network over existing methods.

4.7 Case Study and Demonstration

We develop a BCI system for motion intention recognition and evaluate the proposed models on our real-world BCI dataset. Neuroscan's Grael EEG amplifier and 32-channel EEG cap are used with sampling rate of 256 Hz, which can be seen from Figure 5. Baseline correction, 50 Hz notch filter, and band-pass filter (4-30 Hz) are applied to eliminate baseline drift and noise. Each subject performed motor execution (ME) and motor imagery (MI) to finish four-class intentions, including eye closed, both feet, right fist and left fist open and close. In each recording session, the participants perform each task for 3 seconds, followed by rest for 3 seconds. Every volunteer performs 160 trials, and there are totally 4 male volunteers. Table 4 shows the classification results on our BCI dataset.

Table 4 shows the evaluating results on our BCI dataset. Our method achieves 50.6% classification accuracy of ME and 35.1% classification accuracy of MI on our BCI dataset. Compared with the best baseline model, our methods improve the result by 8.5% classification accuracy of ME and 2.3% classification accuracy of MI, respectively. In summary, the proposed regional attention convolutional neural network is capable of learning spectral-spatial-temporal features and outperforms all those methods.

The proposed method was finally used to develop an EEG based BCI system, as shown in Figure 5. The movement of a virtual ball is controlled by the motion intention of the participant. For example, when the participant executes both feet, right fist and left fist movement, the virtual ball will go straight, turn right and turn left, separately.

5 Conclusion

In this paper, we propose a novel approach to exploit spectralspatial-temporal features for EEG motion intention recognition. Firstly, we propose a novel regional attention convolutional neural network, which learns spectral-spatial-temporal features by feature extraction module, self-attention module, and regional attention module. Secondly, we adopt complex Morlet wavelet convolution and baseline normalization to reveal many task-relevant dynamics in EEG data of different brain regions. Finally, experimental results on two benchmark datasets and our BCI dataset show the effectiveness of our method.

Acknowledgments

This work is supported in part by the National Key R&D Program of China (Grant 2018YFB1307804), the National Natural Science Foundation of China (Grants 91848110 and U1913601), the Strategic Priority Research Program of Chinese Academy of Science (Grant No.XDB32000000), and the Beijing Natural Science Foundation under Grant 4202074.