Feature selection of EEG signals in neuromarketing

Feature extraction

Feature extraction generates the discriminative features of the enhanced signals. In a previous research (Aldayel, Ykhlef & Al-Nafjan, 2021), the extraction of EEG features is described using two algorithms, i.e., discrete wavelet transform (DWT) and power spectral density (PSD). Figure 2 presents the block diagram of feature extraction.

EEG signals were filtered to electrodes (F3, F4, AF3, and AF4) that are relevant to preference recognition. In PSD, four frequency bands: theta (4–8 Hz), alpha (8–12 Hz), beta (12–30 Hz), and gamma (above 30 Hz) were extracted and the average signal power was calculated. In DWT, the EEG signals decomposed using four-level Daubechies (db4) wavelets into detail coefficients (D1, D2, D3, and D4) and an approximation coefficient (A4). The numbers of PSD and DWT features were 2,367 and 840, respectively. In this study, PSD is used for extracting features and the effect of feature selection in neuromarketing is evaluated.

Feature selection

For BCI systems, the following fundamental feature selection approaches can be used: filtering, wrapper, embedded, and hybrid. Wrapper and embedded approaches use a classifier (baseline model) to select a subset of features, unlike filtering, which is model independent (Jenke, Peer & Buss, 2014).

• Filter approaches compute the level of association between each feature and the selected class independent of the classifier used. They can effectively detect irrelevant features, but they cannot eliminate redundancy or dependency between features. The main advantages of filter approaches are scalability, speed, and reduced computational overhead. Examples of filtering include maximum mutual information and mRMR, which is popularly used in EEG-based BCIs (Solorio-Fernández, Carrasco-Ochoa & Martínez-Trinidad, 2020; Jenke, Peer & Buss, 2014).

• Wrapper approaches evaluate the feature subsets using the results of a classification algorithm until the accuracy result is satisfied. They can achieve high accuracy, but due to the high computational cost and time, they select a small subset. RFE is an example of a wrapper approach (Solorio-Fernández, Carrasco-Ochoa & Martínez-Trinidad, 2020; Jenke, Peer & Buss, 2014).

• Embedded approaches combine feature selection and performance evaluation in a particular process, such as decision tree and linear discriminant analysis (LDA).

• Hybrid approaches comprise two phases: filtering to decrease the feature sizes, followed by wrapper to find the optimal subset of features from the remaining features. Hybrid approaches combine the advantages of the filter and wrapper approach to achieve a good trade-off between low computational overhead and increase accuracy in the associated classification task with the selected features (Cai et al., 2018; Solorio-Fernández, Carrasco-Ochoa & Martínez-Trinidad, 2020).

Class separability is an important evaluation criterion in the filter approach. It is determined by measuring the distance between classes, followed by the removal of features that lead to low separability values. Some research design evaluation measurements exist for determining the relevance between features and class (e.g., consistency and relevance measures, ranking and correlation measures, and dependency measures).

Preference indices

On the basis of the literature reviews given in Aldayel, Ykhlef & Al-Nafjan (2020a), the following EEG indices were defined to measure consumers’ responses to marketing stimuli: approach/withdrawal (AW) index, valence, choice index, and effort index. AW index measures the frontal alpha asymmetry, indicating the motivation and desire as higher alpha activation of the left frontal cortex. Equation (10) is used to calculate the difference between the right and left PSD. Touchette’s AW index (Eq. 1) was also used to calculate AW scores by taking the difference between the right and left PSD, divided by their sum (using electrodes F4 and F3) (Touchette & Lee, 2017).

(1) $$\mathrm{T}\mathrm{o}\mathrm{u}\mathrm{c}\mathrm{h}\mathrm{e}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{A}\mathrm{W}\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{x}={\displaystyle \frac{alpha(F4)-alpha(F3)}{alpha(F4)+alpha(F3)}}$$

(2) $$\mathrm{A}\mathrm{W}\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{x}=alpha(AF4,F4)-alpha(AF3,F3)$$

The valence indicates the asymmetrical activation of the frontal hemisphere. It is computed using the following four equations, which were well-explained in Al-Nafjan et al. (2017).

(3) $$\mathrm{V}\mathrm{a}\mathrm{m}\mathrm{v}\mathrm{\_}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}={\displaystyle \frac{beta(AF3,F3)}{alpha(AF3,F3)}-{\displaystyle \frac{beta(AF4,F4)}{alpha(AF4,F4)}}}$$

(4) $$\mathrm{K}\mathrm{i}\mathrm{r}\mathrm{k}\mathrm{\_}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}=\mathrm{l}\mathrm{n}[\mathrm{a}\mathrm{l}\mathrm{p}\mathrm{h}\mathrm{a}(\mathrm{A}\mathrm{F}3,\mathrm{F}3)]-\mathrm{l}\mathrm{n}[\mathrm{a}\mathrm{l}\mathrm{p}\mathrm{h}\mathrm{a}(\mathrm{A}\mathrm{F}4,\mathrm{F}4)]$$

(5) $$\mathrm{R}\mathrm{a}\mathrm{m}12\mathrm{\_}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}=\mathrm{a}\mathrm{l}\mathrm{p}\mathrm{h}\mathrm{a}(\mathrm{F}4)-\mathrm{b}\mathrm{e}\mathrm{t}\mathrm{a}(\mathrm{F}3)$$

(6) $$\mathrm{R}\mathrm{a}\mathrm{m}15\mathrm{\_}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}={\displaystyle \frac{\mathrm{a}\mathrm{l}\mathrm{p}\mathrm{h}\mathrm{a}(\mathrm{F}4)}{\mathrm{b}\mathrm{e}\mathrm{t}\mathrm{a}(\mathrm{F}4)}-{\displaystyle \frac{\mathrm{a}\mathrm{l}\mathrm{p}\mathrm{h}\mathrm{a}(\mathrm{F}3)}{\mathrm{b}\mathrm{e}\mathrm{t}\mathrm{a}(\mathrm{F}3)}}}$$

The choice index measures choice possibility in decision-making from the frontal asymmetry of beta and gamma. Ramsoy’s equation (Ramsøy et al., 2018), as shown in Eq. (8), was used to calculate the choice index for each band individually (gamma and beta) using electrodes AF3 and AF4.

(7) $$\mathrm{C}\mathrm{h}\mathrm{o}\mathrm{i}\mathrm{c}\mathrm{e}\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{x}(\mathrm{g}\mathrm{a}\mathrm{m}\mathrm{m}\mathrm{a})={\displaystyle \frac{log(gamma(AF3))-log(gamma(AF4))}{log(gamma(AF3))+log(gamma(AF4))}}$$

(8) $$\mathrm{C}\mathrm{h}\mathrm{o}\mathrm{i}\mathrm{c}\mathrm{e}\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{x}(\mathrm{b}\mathrm{e}\mathrm{t}\mathrm{a})={\displaystyle \frac{log(beta(AF3))-logbeta((AF4))}{log(beta(AF3))+log(beta(AF4))}}$$

The effort index indicates the cognitive and activity level of the theta in frontal cortex. The following equations were used to calculate the effort index:

(9) $$\mathrm{E}\mathrm{f}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{t}\mathrm{I}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{x}\mathrm{\_}1={\displaystyle \frac{theta(F4)-theta(F3)}{theta(F4)+theta(F3)}}$$

(10) $$\mathrm{E}\mathrm{f}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{t}\mathrm{I}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{x}\mathrm{\_}2=theta(AF4,F4)-theta(AF3,F3)$$

Moreover, a combination of preference indicators with different time—frequency analyses was used to measure EEG-based preference indices.

mRMR

Peng, Long & Ding (2005) proposed the mRMR filter method, which selects features on the basis of relevance and redundancy measures. This method initially involves the selection of features with the largest correlation of class label and the least correlation of other features. The statistical correlation is computed using mutual information I, thereby ensuring that the joint distribution can optimize the following two criteria simultaneously: (1) maximum relevance/dependence D between features and the class label $I\left({\mathbf{x}}_a;\mathbf{y}\right)$ and (2) minimum redundancy R of features $I\left({\mathbf{x}}_a;{\mathbf{x}}_b\right)$, which maximizes classification accuracy (Peng, Long & Ding, 2005; Jenke, Peer & Buss, 2014). Assuming a chosen set ${\text{𝒫}}_q$ of q features, the feature is selected by optimizing the combined criterion D − R:

(11) $$\underset{{\mathbf{x}}_a\in \text{𝒳}-{\text{𝒫}}_q}{max}\left[I\left({\mathbf{x}}_a;\mathbf{y}\right)-{\displaystyle \frac{1}{q}\sum _{{\mathbf{x}}_b\in {\text{𝒫}}_q}I\left({\mathbf{x}}_a;{\mathbf{x}}_b\right)}\right]$$

mRMR can be effectively integrated with wrappers to form a hybrid feature selection approach that can reduce computational overhead with greater speed and accuracy (Peng, Long & Ding, 2005; Jenke, Peer & Buss, 2014). Many researchers have found that mRMR is an effective feature selection approach for EEG signals (Atkinson & Campos, 2016; Moon et al., 2013; Jenke, Peer & Buss, 2014). mRMR-based feature selection improves the accuracy of the kernel classifier for emotion recognition. Moreover, another study of preference recognition showed that mRMR can enhance classifier accuracies with band power features (Moon et al., 2013).

RFE

RFE is a wrapper method that is widely used for feature selection. It works by reducing features recursively and constructing a classification model on the remaining features. The algorithm rearranges the ranking of candidate feature subsets with corresponding classification accuracies for each iteration. The classification accuracy is used to select a combination of features that contributed the most to class label prediction (Cai et al., 2018; Jiang et al., 2020). RF classifier is used in the RF-RFE method to measure the importance of features and to build a classifier with high importance scores. Figure 3 illustrates the flowchart of RF-RFE 3 (Chen et al., 2018).

ReliefF

Relief-based algorithms are widely used for their simplicity and efficiency. The ReliefF technique is a standard filtering approach for feature selection representing an extension of the original Relief algorithm. It is known as the best practice of Relief-based algorithms (Jenke, Peer & Buss, 2014). ReliefF assigns weights to each feature based on its ability to separate class labels. ReliefF relies on some neighbor’s k to specify the k-nearest hits and misses to avoid the sensitivity of feature dependencies. Multiple classes are considered to search the k-nearest elements that are weighted by their earlier probabilities. This change increases weight estimate reliability, particularly in noisy problems. The efficiency of ReliefF is attributed to the fact that it is “intended as a screener to identify a subset of features that cannot be the smallest and can still include some irrelevant and redundant features, but that is small enough to use with more refined approaches in a detailed analysis” (Urbanowicz et al., 2018; Jenke, Peer & Buss, 2014).

PCA

Recently, unsupervised feature selection approaches have piqued research interest owing to their ability to select related features without depending on the class label. This makes them unbiased, allowing them to work effectively even in the absence of prior knowledge. Thus, they decrease the probability of overfitting (Solorio-Fernández, Carrasco-Ochoa & Martínez-Trinidad, 2020).

PCA is one of the most common unsupervised feature selection methods used for dimensionality reduction. It applies statistical methods to transform a set of interrelated measurements into a set of linearly unrelated principal components. The significance of PCA relates to its ability to reduce dimensionality without information loss while considering the complexity of signal extraction and classification. PCA extracts useful signals from the time series data of EEG signals. It compresses EEG signals into unrelated components for signal preprocessing and feature selection, thereby reducing the noise in the signal separation. In turn, EEG signals are reconstructed after separating noise. When PCA detects patterns in a signal, it can be imagined as involving the rotation of coordinate axes along with the combination of time points. The principal components are the components that have a maximum variance (Xie & Oniga, 2020; Torres et al., 2020).

Dataset

A public benchmark neuromarketing dataset (Yadava et al., 2017) that has been utilized in BCI-based neuromarketing experiments is used in this study. The Emotiv EPOC+ headset was used for capturing the EEG signals. A total of 25 male participants, aged 18–38 years, have recorded their 42 trials using the visual stimulus of 14 products, each having three variations (14 × 3 = 42). While the participants watched products on a computer screen, their EEG signals were recorded. Thus, 1,050 (= 42 × 25) records were logged for all participants. EEG signals were downsampled to 128 Hz and signal preprocessing (bandpass filter 4.0–45.0 Hz, independent component analysis, and a Savitzky–Golay filter) were performed to eliminate irrelevant features and artifacts. The features were collected from 14 electrodes located on “AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4”. For preference recognition, four relevant electrodes were selected (F3, F4, AF3, and AF4). The selection of these electrodes was based on preference mapping with brain areas as explained in a previous research (Aldayel, Ykhlef & Al-Nafjan, 2020a). Responses in the form of either “like” or “dislike” were gathered from each participant watching a product, and EEG data were logged simultaneously. After each picture was displayed, the user’s preferred choice was recorded. The brain activity map in Fig. 4 shows the disruptions of brain activity in the two preference states “like” and “dislike”.

For validation, holdout validation was used to split the dataset: 80% for training and 20% for testing. For feature extraction, the fast Fourier transform method was used to extract PSD features. The dimensions of PSD features include 257 for each selected channel. Moreover, the sum and average of PSD feature across all four channels were combined. The average power of PSD was calculated using Morlet time—frequency transform into (61) features for each selected channel. In addition, (65) spectrogram Hanning features were extracted. The total numbers of PSD features were 2,367.

Feature selection based on importance

The top 10 features in RF-RFE were selected for their importance to the preference state were measured using RF, mRMR, and ReliefF. The mRMR and ReliefF algorithms were implemented using the fscmrmr and relieff functions in MATLAB. Moreover, the feature importance property in the RF algorithm implemented in the scikit-learn package in Python was used to calculate feature importance in RF-RFE. The variance between the top 10 most important features was indicated via the analysis of variance (ANOVA) F-test. Finally, the correlation between all of the top features and class labels was measured using a correlation matrix with a heatmap in Python’s seaborn package. Figures 5 and 6 show the top 10 feature rankings in mRMR and ReliefF, respectively.

Feature selection and classification algorithms

This study aims to detect two preference states (“like” or “dislike”) in EEG neuromarketing data. Hence, intelligent classification algorithms were deployed to effectively mirror the preferences of the subjects. A DNN classifier is proposed and its performance is compared to the KNN, RF, LDA, and SVM classifiers. As such, four classifiers were used to discover the optimal preference index and a well-matched classifier marked with the highest accuracy: DNN, KNN, LDA, RF, and SVM. The default hyperparameters for the KNN, RF, and SVM algorithms were used in the scikit-learn package to adjust the following hyperparameters.

• KNN: Adjusted the number of neighbors to 1.

• RF: Adjusted the number of trees in the forest to 500, which were all processed in parallel.

• LDA: Set the number of components to 1.

• SVM: Used the kernel of RBF.

Feature importance results

Figures 5 and 6 present the top 10 feature rankings using the mRMR and ReliefF algorithms, respectively. Figure 7 presents the top 10 important features in the RF algorithm that were recursively considered in RFE. Since PCA separates components with the maximum variance; the ANOVA F-test illustrates the variance between the top 10 important features related to class labels (Fig. 8). The F-value scores check if the average for each combination of numerical features by the class label is significantly different.

Correlation is a measure of the positive or negative linear relationship between any two features. A heatmap is used to illuminate the correlation between the top 10 important features. The correlation matrix in the heatmap (Fig. 9) illustrates the values of Kendall’s tau correlation coefficient between the top 10 important features and the class labels. Kendall’s rank correlation coefficient measures similarities of orderings of data that are ranked by their quantities. The most interesting features in terms of their correlation scores to the class label were Vamv valence (0.70), Kirk valence (0.53), AW (0.50), and effort index 2 (0.54). Notably, Vamv valence and Kirk valence were the top features in mRMR, ReliefF, RF, ANOVA F-measure, and Kendall’s rank correlation coefficients. The third most important feature was effort index 2 in RF, ANOVA F-measure, and Kendall’s rank correlation coefficient. This is consistent with the neuroscientific interpretation of the dataset owners (Yadava et al., 2017). They achieved the highest accuracy rate of 70.33% with the theta band, which was represented as effort index 2 in this research.

Classification results

Since the results of feature selection depend on the classification performance, the classification performance before and after applying the feature selection was evaluated. The study aims to evaluate the effect of applying feature selection to enhance the identification of preference states (i.e., “like” and “dislike”) in the EEG neuromarketing data.

The EEG signal data was divided (80% training and 20% for testing). Classifiers that efficiently replicate the subjects’ preferences were used. The extracted features (PSD and preference indices) were fed into five classifiers (LDA, SVM, RF, DNN, and KNN).

In comparison with other loss functions, the DNN result with the hinge loss function achieved the highest accuracy. The performances of these classifiers were compared with and without different feature selection methods (REF, ReliefF, PCA, feature importance, and mRMR), as presented in Table 1. Additionally, the number of selected features set to 10 in ReliefF, feature importance, and mRMR. In REF and feature importance, we also set the number of features to 30. In PCA, the features were reduced from 2,367 to 1,050. Figures 10 and 11 show the results from the viewpoint of feature selection. The figure demonstrates the effect of different feature selection algorithms on the accuracy and F-measure of the LDA, SVM, RF, DNN, and KNN classifiers.

For all classifiers, the best results were achieved with mRMR and ReliefF. The KNN, LDA, and SVM classifiers yielded enhanced accuracies of 94%, 95%, and 97%, respectively, whereas DNN and RF achieved the highest accuracies of 99% and 100%, respectively. Similar results were achieved with the feature importance method, but all classifiers, except DNN, yielded better accuracies when the number of selected features was set to 10. DNN achieved a better accuracy at 98% with 30 features. The reason for this is that DNN works better with a large dataset. PCA did not yield enhanced accuracy compared with the other feature selection methods.

In summary, the mRMR feature selection algorithm achieved the best classification results (i.e., in terms of accuracy, recall, and precision) at 100% with RF and 99% with DNN (hinge function). RF reached the best results across all feature selection methods. The LDA, SVM, and KNN classifiers achieved an approximately 20% increase in accuracy after applying feature selection. Generally, feature selection was found to improve the performance of all classifiers.