Emotion Recognition via fMRI-Derived Brain States using Deep Neural Network

IntroductionEmotion is one of the most important aspects of human life, and understanding them is challenging, especially for computers but the use of reliable measures such as blood oxygenation level-dependent (BOLD) signals measured by functional magnetic resonance imaging (fMRI) could help us better understand them.MethodsWe applied Long Short-Term Memory (LSTM) neural network [۱] to fMRI-derived beta-series matrices of dimension ۹۰×۳۷۱۳۰ corresponding to ۹۰ stimuli and ۳۷۱۳۰ gray-matter voxels to predict the normative valence and arousal scores (in ۹-point Likert scale) of affective stimuli [۲]. We also used our model as a classifier to classify each stimulus responses into two categories: high/low arousal and positive/negative valence. The proposed deep learning neural network model consists of two fully connected LSTM layers, a dropout layer, and a dense layer. Owing to deep networks’ overfitting nature, both elastic net and dropout regularization approaches were used to further enhance the network’s generalization capabilities. The model was trained on ۸۰% of each subject's data and tested on ۲۰% of the remaining data based on ۵-fold cross-validation scheme.ResultsFour different binary classifiers were trained on the mentioned feature matrices and the algorithms were applied to each subject separately. Then, the comparison between the average accuracy of all subjects provided by each model is reported in figure ۱. After performing chi-square feature selection, the classification accuracies improved from ۷۰% to ۸۰% for valence, and from ۷۳% to ۸۱% for arousal. The proposed model was also trained as a regression model. Pearson’s Correlation Coefficient results are also summarized in table ۱.ConclusionResults indicate that deep learning, should there be enough data, is a promising choice for emotion recognition as features can be learned directly from raw data. As stated, LSTM achieves higher average accuracy over subjects compared to other traditional methods.