EEG-based Feature Space for Supporting Deep Neural Networks in Image Classification

Decoding human brain activity evoked by visual stimuli has always been an interesting topic in cognitive neuroscience. In this paper, electroencephalographic (EEG) signals are employed to enhance the image classification accuracy and indirectly define the targets of the images, by fusing EEG-based regressed features and the features extracted from a deep neural network. To this end, we proposed an EEG encoder that included one Long Short-Term Memory (LSTM) layer followed by one convolutional layer to extract the representative features from EEG signals. Then, these EEG-based features are fused with those from a deep neural network (DNN) which is fed directly by the corresponding images. These fused feature vectors and predicted labels from the proposed encoder are employed to train an SVM-based classifier due to its generalization ability. In the test phase, the DNN-based visual feature is projected onto the proper EEG-based feature space via a regressor, and the fused feature vector is obtained and applied to the SVM. Evaluating the proposed model on the ImageNet-EEG dataset, which includes ۴۰ classes of images, shows that the proposed encoder reaches an average accuracy of ۹۹.۳۵% for classifying EEG signals. The proposed human brain-guided system also improved image classification accuracy by over ۱۰% compared to the deep neural network.

کلیدواژه ها:

Image classification ، Brain-guided System ، EEG signals ، Deep Neural Network ، Feature fusion