Classifying imbalanced data in distance-based feature space

Abstract

Class imbalance is a significant issue in practical classification problems. Important countermeasures, such as re-sampling, instance-weighting, and cost-sensitive learning have been developed, but there are limitations as well as advantages to respective approaches. The synthetic re-sampling methods have wide applicability, but require a vector representation to generate additional instances. The instance-based methods can be applied to distance space data, but are not tractable with regard to a global objective. The cost-sensitive learning can minimize the expected cost given the costs of error, but generally does not extend to nonlinear measures, such as F-measure and area under the curve. In order to address the above shortcomings, this paper proposes a nearest neighbor classification model which employs a class-wise weighting scheme to counteract the class imbalance and a convex optimization technique to learn its weight parameters. As a result, the proposed model maintains the simple instance-based rule for prediction, yet retains a mathematical support for learning to maximize a nonlinear performance measure over the training set. An empirical study is conducted to evaluate the performance of the proposed algorithm on the imbalanced distance space data and make comparison with existing methods.

Appendices

Appendix 1: Structural SVM learning

A structural classifier addresses a problem with multivariate input or output variables that are structured or dependent [10, 11]. The dependency is captured in the max-margin learning formulation using a feature function and an optimizer. The feature function ${\varPsi }$ generates an arbitrary representation from a pair of input/output values $(\mathbf {x},y)$. A discriminant function $F:X\times {Y}\rightarrow \mathbb {R}$ is then defined as an inner product of a weight parameter $\mathbf {w}$ and ${\varPsi }$

$$\begin{aligned} F(\mathbf {x},y;\mathbf {w})=\langle \mathbf {w},{\varPsi }(\mathbf {x},y)\rangle \end{aligned}$$

(30)

From F and the optimizer that selects a class over Y, the decision function $f(\mathbf {x};\mathbf {w})$ is defined as

$$\begin{aligned} f(\mathbf {x};\mathbf {w})=\mathop {\arg \max }\limits _{y}F(\mathbf {x},y;\mathbf {w}) \end{aligned}$$

(31)

A typical example of the structural classifier is the multiclass SVM described in [11]. Let $\{c_j\}_{j=1}^n$ denote the class values and ${\varLambda }(c_j)=\mathbf {e}_j$ the binary encoding of $c_j$, i.e., a unit vector whose jth element is 1. The feature function ${\varPsi }$ is defined as

$$\begin{aligned} {\varPsi }(\mathbf {x},y)=\mathbf {x}\otimes {\varLambda }(y) \end{aligned}$$

(32)

where $\otimes$ denotes the tensor product.

Substituting the feature function (32) and a stack of vectors $\mathbf {w}=[\mathbf {w}'_1,\ldots ,\mathbf {w}'_p]$ as parameters into (30), the discriminant function produces a set of margins for respective classes. Subsequently, the class predicted by (31) is the class with the largest margin.

Training of the structural classifier is formulated in a quadratic programming problem [10]. Given the training samples $\{(\mathbf {x}_i,y_i)\}_{i=1}^\eta$,

Problem 2

$$\begin{aligned} \mathop {\arg \min }\limits _{\mathbf {w},\xi \ge 0}\frac{1}{2}\Vert \mathbf {w}\Vert ^2 +C\xi \end{aligned}$$

subject to $\forall {i},\forall {y}\ne {y}_i$,

$$\begin{aligned} \langle \mathbf {w},{\varPsi }(\mathbf {x}_i,y_i)\rangle -\langle \mathbf {w},{\varPsi }(\mathbf {x}_i,y)\rangle \ge 1-\xi \end{aligned}$$

(33)

An approximated solution of Problem 2 can be obtained by a cutting-plane algorithm [22].

Appendix 2: Multiclass formulation of SNN classifier

This section describes the formulation of SNN classifier learning with multiple majority and minority classes. Similar to the binary classification problem described in Sect. 4.2, the main intuition is to maximize the margin between the correct input and all other cases through the constrained minimization of the $\ell 2$-norm.

Let $\fancyscript{M}=\{i\}_{i=1}^m$ and $\fancyscript{N}=\{m+j\}_{j=1}^n$, denote the class values of the majority and minority classes, respectively. Note that the class values of $\fancyscript{N}$ differ from Sect. 4.2. For mathematical convenience, a matrix notation W is introduced to represent the weights considered in the selection of the nearest neighbors among different classes. The relation between W and the weight vector $\mathbf {w}$ is defined as follows:

$$\begin{aligned} W=\left[ \begin{array}{l@{\qquad }l} \begin{array}{l@{\quad }l@{\quad }l} 1&{}\cdots &{}1\\ \vdots &{}\ddots &{}\vdots \\ 1&{}\cdots &{}1\\ \end{array} &{} M\\ M' &{} \begin{array}{l@{\quad }l@{\quad }l} 1&{}\cdots &{}1\\ \vdots &{}\ddots &{}\vdots \\ 1&{}\cdots &{}1\\ \end{array} \end{array} \right] _{(m+n)\times (m+n)} \end{aligned}$$

(34)

where

$$\begin{aligned} M=\left[ \frac{w_i}{w_j}\right] _{m\times {n}}, \quad M'=\left[ \frac{w_j}{w_i}\right] _{n\times {m}} \end{aligned}$$

W can be computed automatically from each $\mathbf {w}$ and is not shown in the main content to avoid the overuse of notations.

Let us define the decision function of an individual input as follows:

$$\begin{aligned} f'(X;W)=\mathop {\arg \max }\limits _{y\in \fancyscript{M}\cup \fancyscript{N}}W^\top (\mathbf {u}(y)\odot \mathbf {d}) \end{aligned}$$

(35)

where $\odot$ denotes the Hadamard product and $\mathbf {u}(y)$ the unit vector which has 1 at yth position.

In (35), the yth row of W is considered in choosing the nearest neighbor. From the definition of W, the weights are equivalent among the minority classes if $y\in \fancyscript{M}$. Otherwise, i.e., $y\in \fancyscript{N}$, the weights are equivalent among the majority classes.

The decision functions for the structural input is written as a summation of $f'$,

$$\begin{aligned} f(X;{W})=\mathop {\arg \max }\limits _Y\sum _h{W}^\top \left( \mathbf {u}(y_h)\odot \mathbf {d}(x_h)\right) \end{aligned}$$

(36)

Based on (36), let us define the feature function ${\varPsi }$ as follows

$$\begin{aligned} {\varPsi }(X,Y)=\sum _{h=1}^\eta \mathbf {u}(y_h)\odot \mathbf {d}(x_h) \end{aligned}$$

and the discriminative function F as a matrix product of W and ${\varPsi }$.

$$\begin{aligned} F(X,Y;{W})={W}^\top {\varPsi }(X,Y) \end{aligned}$$

Using ${\varPsi }$ and F, (36) is rewritten in the same form as (26)

$$\begin{aligned} f(X;{W})=\mathop {\arg \max }\limits _{Y}F(X,Y;\mathbf {w}) \end{aligned}$$

(37)

From (37), the constrained $\ell 2$-norm minimization problem is formulated as follows.

Problem 3

$$\begin{aligned} \mathop {\min }\limits _{\mathbf {w},\xi \ge 0}\frac{1}{2}\left\| \mathbf {w}\right\| ^2+C\xi \end{aligned}$$

subject to $\forall \mathbf {z}\in (\fancyscript{M}\cup \fancyscript{N})^\eta {\setminus }\{\mathbf {y}\}$

$$\begin{aligned} W^\top {\varPsi }(\mathbf {z},X)-{W}^\top {\varPsi }(Y,X)\ge {\varDelta }(\mathbf {z},Y)-\xi \end{aligned}$$

(38)

Problem 3 can be solved by the cutting-plane method described in Sect. 4.