Building a novel classifier based on teaching learning based optimization and radial basis function neural networks for non-imputed database with irrelevant features

2.1 Imputation of missing values and feature selection

The problem of classification is basically the foundation of dividing the feature space into sections, one section for each category of inputs. Classifiers are usually, designed with labeled data, which is sometime referred to as supervised classification. In general, classification with missing data and irrelevant features focuses on three distinct tasks: handling missing values [1] (i.e., imputing values), feature selection, and pattern classification. Let D = [x_ij]_Nxd, where i = 1, 2,…, N_, and j = 1,2,…,d, is the dataset containing N samples and d features. In D, each sample is assigned a class label from the set C = {c₁, c₂,..,c_M}, where |C|= M. Let each x_ij, be represented as a tuple (x_ij, y_ij), in which y_ij can take only two values either 0 or 1. If the value of v_ij = 0, then its associated x_ij value is missing, otherwise present. Input data has quantitative and qualitative variables. Quantitative or continuous data is measured on a numerical scale. Non-numerical (i.e., colors, names, opinions) is called qualitative data, which can be discrete or categorical. The overall goal of handling missing value is to map the value of y_ij from 0 to 1 by substituting an appropriate value of x_ij with less bias.

Alongside feature selection problem is defined as to select a subset of features from the given set of features, thereby the dataset is mapped from (x_ij)_Nxd to (x_ij)_Nxk, where k ≪ d. With this intention, filter method is selecting the most relevant features, however, a predefined quality measure is necessary to establish the level of relevance of the features. Filter method is not able to identify correlation among the features simultaneously. Unlike filter, wrapper is able to address correlation among features because it uses the performance of the classifier to optimize the subset. This also led towards problem of intractability. Moreover, this method has the additional cost of reconstructing the classifier with modified feature subset. Hence to avoid these issues, a filter like algorithm known as Relief method is employed here.

2.2 Radial basis function networks

The RBF network [8] is a topology having three layers: an input, a hidden, and a linear output layer (see Figure 1). The input can be modeled as n-dimensional input vector. The hidden layer implements a radial activation function and that carry out a non-linear transformation from the input space to the hidden space. The center and width are two parameters associated for each hidden node. Usually, a nonlinear transformation from input to the hidden space is made based on Gaussian kernel as described in Eq. (1).

The radial basis function is so named because the value of the function is same for all points which are at the same distance from the center.

In literature, radial basis function networks [6] have many extensive uses, including classification, time series prediction, function approximation, etc. Training RBF networks is normally faster than training multi-layer perceptron (MLP) networks. Training of RBF network [9,11] involves two steps: (1) the kernel parameters of the hidden neurons are determined by an unsupervised method or heuristic method; (2) The weights of the output-layer are determined by pseudo-inverse method.

2.3 Teaching learning based optimization

Teaching learning based optimization is one of the population based nature inspired algorithms introduced by Rao et al. [6,9]. This is inspired purely from the natural phenomena of teaching-learning process that motivated by a teacher on the output of learners within a classroom environment, where learners first obtain knowledge from teacher and subsequently from classmates. In the first phase, a teacher imparts knowledge directly to his/her students. In practice, the possibility of a teacher’s teaching being successful, is distributed under Gaussian law. Overall, how much knowledge is transferred to a student depends not only on his/her teacher but also interactions among the students through peer learning. A basic algorithm of TLBO is presented below.

Opens in a new window.

3.1 Missing value imputation using least-square estimator

In this phase, we estimate the missing value from D by formulating a matrix A, where all the attribute values are known. In the least-square problem, the output of a model is given by the linearly parameterized expression,

If the target system has q outputs, expressed as y=[y1,…yq]T with q > 1 then we have a set of linear equations in matrix form, AΘ + E = Y, where A is an mxn matrix as given below:

and Θ is an n×q unknown parameter matrix:

is an m×q output matrix with yij denoting the jth output value in the ith data pairs.

After getting the value of Θ, we will continue imputing all the values in the data set D.

3.2 Relief algorithm for feature selection

In this second phase of our work, we discuss Relief algorithm inspired by instance based learning. It is an filter method algorithm for individual feature selection. It calculates a proxy statistics for each feature that can be used to estimate the feature quality or relevance to the target concept. The pseudocode of this method is given below.

Opens in a new window.

3.3 Improved TLBO based RBFN

In third phase, we are building a RBFN classifier which is trained by TLBO and improved TLBO. First we will provide a detailed introduction to the improved TLBO and then the improved TLBO + RBFN network is developed with the aim of achieving better classification accuracy.

3.3.1 Improved TLBO (iTLBO)

In the canonical TLBO, during the learning phase the learner is exposed to the entire population of the class. However, it has been realized that if the learner is restricted with a peer team instead of all individuals of the population then he/she can raise his/her level of acquiring knowledge. With this idea, we are introducing a neighborhood structure of learners as peer learners group for making a learner to learn. Hence, in the learner phase, we have adopted a square topology as peer learners group for a learner. That means a student will not only acquire knowledge from the best of all individuals (i.e., teacher) but also he/she improves his/her standard from his neighborhood of fellow learners. In that context, the learner phase of TLBO has been modified as given below.

Opens in a new window.

Here the nearest_neighbor( ) will find out a group of peer learners for a learner. The size of the neighborhood can be treated as a parameter for learner phase. Alongside, we have also made the teaching parameter (T_F) adaptive by considering the individual fitness value and population diversity. Recall that the teaching factor decides the value of mean to be changed. In the canonical TLBO, the value of T_F is either 1 or 2 thereby learners learn nothing from the teacher or learn all the things from the leaner. But in real practice, the value of T_F may be between 1 and 2 include both. Hence to make this idea fruitful, the fitness variable is selected as inputs to choose T_F. BS is containing the global best solution denoted as X^k_g found so far i.e. up to kth iteration, which is just a position for one individual, corresponding to the best fitness F^k_g. So the global best solution fitness differentials between kth and k-1th can be defined as:

(4)ΔFk=|FSBk−FSBk−1|.

Now, we can give definition for function of convergence speed as follows:

(5)Cs=ΔFk/Δ,

where Δ1=max{ΔF1, ΔF2, …ΔFk} Eq. (5) can calculate the convergence speed, which is less than or equal to 1.

In evolution process of TLBO, population diversity is a major factor. For computing the diversity of the population, standard deviation of the individual fitness values of population can be used. In this paper, we present a new strategy for calculating population position diversity by fitness value. The population position diversity can be obtained by using deviation ideology approach defined in Eqs. (6)–(8).

(6)Favgk=1/|P|∑i=1|P|Fk(i),

(7)Δ2=max{|Fk(1)−Favgk|, …,|Fk(|P|)−Favgk}|,

(8)σ2=1/|P|∑i=1|P|(Fk(i)−FavgkΔ2)2,

where Favgk is the average population fitness for current kth iteration; Fk(i) stands for ith individual fitness; Δ2 stands for normalization factor; |P| is the population size, σ2 represents the population diversity. It is evident that larger the σ2, the larger is the population diversity.

To improve adaptive teaching factor T_F, we use the index C_S for representing the convergence speed with respect to the best solution fitness found so far in current iteration, and the index σ2 to represent diversity with regards to the population fitness deviation. Hence we can compute T_F by C_S and σ2 adaptively, as follows:

(9)TF=α.CS+β.σ2+1,

where α, β are factors; σ2 and C_S are less than or equal to 1 and greater than 0, so 1≤TF≤α+β+1, Rao et al. [6] suggested that the value of T_F can be either 1 or 2. Hence, we set α+β+1≤2. The proposed method of adaptive teaching factor (T_F) is applied for better local searching ability that improves the accuracy and convergence speed.

3.3.2 iTLBO + RBFN

This section describes the iTLBO + RBFN which can adjust the network parameters during the training process. In the initialization stage, let the position of the ith individual be represented as shown in Figure 3. RBFNs mainly depend on center and width of the kernel in addition to weights and bias. However, here, we just encode the centers, widths, and bias into an individual for stochastic search using iTLBO.

Suppose the maximum number of kernel nodes is set to K_max, then the structure of the individual is represented as follows (c.f., Figure 3):

In other words, each individual has three constituent parts such as center, width, and bias. The length of the individual is 2K_max + 1.

The fitness function which is used to guide the search process is defined in equation (10).

(10)f(x)=1N∑i=1N(ti−Φ^(xi→))2

where, N is the total number of training instances, t_i is the actual output and Φ^(xi→) is the estimated output of RBFNs. Initially, the centers, widths, and bias are computed using training vectors, the weight is computed using pseudo-inverse method.

(11)Y=WΦ⇒W=(ΦTΦ)−1ΦTY

Opens in a new window.

4.1 Description of datasets and parameters

The datasets used in this work were obtained from the UCI machine learning repository [10]. Seven datasets have been chosen to validate the proposed method i.e., iTLBO + RBFN. The details about the seven datasets are given in Table 1. The algorithmic parameters like population size, number of iterations, etc are fixed based on empirical analysis as follows.

The size of the population is equal to 100, number of iterations fixed at 300, size of the neighborhood is restricted with 10% of population size, and the value of T_F has been adapted as per suggestions given in sub-section 3.3.1 with α value from (0, 1) and β value from (0, 1). The parameters of multi-layer perceptron (MLP) along with training algorithms and Simple Logistic are defined as prescribed in [3].

4.2 Results and analysis

The average results of the experiment obtained from 10 fold cross validation of 30 independent runs are given in Tables 2–7.

From Table 2 it is found that for 7 different datasets iTLBO + RBFN gives better accuracy than TLBO + RBFN, MLP, and Simple Logistic. To support the above results of TLBO + RBFN, statistical analysis based on the measures derived through confusion matrix is presented in Tables 3 and 4.

From the Statistical analysis it can be observed that the calculated Kappa-values for TLBO + RBFN with feature selection are much better than TLBO + RBFN without feature selection.