Random forests with parametric entropy-based information gains for classification and regression problems

Classification task

The target function for selecting the splitting of each internal node is usually based on Shannon entropy. In this work, we consider three target functions based on the following parametric entropies: Renyi entropy, Tsallis entropy, and Sharma-Mittal entropy. Analogously to Eq. (1), Renyi entropy-based information gain for node j can be formulated as follows: (5)${\displaystyle I_j=H\left(S_j\right)-\frac{\left|S_j^L\right|}{\left|S_j\right|}H\left(S_j^L\right)-\frac{\left|S_j^R\right|}{\left|S_j\right|}H\left(S_j^R\right),H\left(S_j\right)=\frac{1}{1-\alpha }log\left(\sum _{c\in C}{\left(p\left(c\right)\right)}^{\alpha }\right).}$ In Eq. (5), H(⋅) is Renyi entropy with parameter α, C is the set of classes, c is a class label.

Tsallis entropy-based information gain is calculated as follows: (6)${\displaystyle I_j=H\left(S_j\right)-\frac{\left|S_j^L\right|}{\left|S_j\right|}H\left(S_j^L\right)-\frac{\left|S_j^R\right|}{\left|S_j\right|}H\left(S_j^R\right),H\left(S_j\right)=\frac{1}{\beta -1}\left(1-\sum _{c\in C}{\left(p\left(c\right)\right)}^{\beta }\right).}$ In equation Eq. (6), H(⋅) refers to Tsallis entropy with parameter β.

Finally, Sharma-Mittal entropy-based information gain is expressed as follows: (7)${\displaystyle I_j=H\left(S_j\right)-\frac{\left|S_j^L\right|}{\left|S_j\right|}H\left(S_j^L\right)-\frac{\left|S_j^R\right|}{\left|S_j\right|}H\left(S_j^R\right),H\left(S_j\right)=\frac{1}{1-\beta }\left({\left(\sum _{c\in C}{\left(p\left(c\right)\right)}^{\alpha }\right)}^{\frac{1-\beta }{1-\alpha }}-1\right).}$ In Eq. (7), H(⋅) is Sharma-Mittal entropy with parameters α and β. Let us notice that Sharma-Mittal entropy becomes Renyi entropy if β → 1 and Tsallis entropy if α → β (Akturk, Bagci & Sever, 2007). To build decision trees for the random forest, one has to maximize the chosen target function to find the best split for each internal node. Then, each leaf node makes a prediction based on the majority principle, i.e., the class is chosen to which the largest number of training data points corresponds.

Regression task

Let us briefly discuss some properties of multiple linear regression with Gaussian noise, which will then be used in the formulation of information gain. Let us consider the equation of such a regression: y = a₀ + a₁x₁ + ... + a_px_p + ϵ, p is the number of factors, ϵ ∼ N(0, σ²) and noise is independent between different measurements. Then y|x ∼ N(x^Ta, σ²), where $x=\left(\begin{array}{c}\hfill 1\hfill \\ \hfill x_1\hfill \\ \hfill ⋮\hfill \\ \hfill x_p\hfill \end{array}\right)$and $a=\left(\begin{array}{c}\hfill a_0\hfill \\ \hfill a_1\hfill \\ \hfill ⋮\hfill \\ \hfill a_p\hfill \end{array}\right)$. In matrix form, it can be expressed as follows: Y = X⋅a + ϵ, where ϵ ∼ MVN(0, σ²I), $X=\left(\begin{array}{ccccc}\hfill 1\hfill & \hfill x_{11}\hfill & \hfill x_{12}\hfill & \hfill ...\hfill & \hfill x_{1p}\hfill \\ \hfill ...\hfill & \hfill ...\hfill & \hfill ...\hfill & \hfill ...\hfill & \hfill ...\hfill \\ \hfill 1\hfill & \hfill x_{n1}\hfill & \hfill x_{n2}\hfill & \hfill ...\hfill & \hfill x_{np}\hfill \end{array}\right)$, $Y=\left(\begin{array}{c}\hfill y_1\hfill \\ \hfill ⋮\hfill \\ \hfill y_n\hfill \end{array}\right)$, n is the number of observations. The estimations of vector a and parameter σ² can be obtained in the following way: (8)${\displaystyle \hat{a}={\left(X^{T}X\right)}^{-1}{X}^{T}Y,{\hat{\sigma }}^2=\frac{1}{n}{h}^{T}h,\text{where}h=Y-X\cdot \hat{a}.}$

In the framework of a decision tree, let us define a probabilistic linear predictor (Criminisi, Shotton & Konukoglu, 2012) for node j as $d_j\left(x\right)=x^T{\hat{a}}_j$, where ${\hat{a}}_j$ is calculated based on the observations fallen into node j according to Eq. (8). Further, let us introduce four types of information gain based on classical Shannon entropy and three parametric entropies taking into account the introduced probabilistic linear predictor. First, Shannon entropy-based information gain for node j can be determined as follows: (9)${\displaystyle I_j=H\left(S_j\right)-\frac{\left|S_j^L\right|}{\left|S_j\right|}H\left(S_j^L\right)-\frac{\left|S_j^R\right|}{\left|S_j\right|}H\left(S_j^R\right),H\left(S_j\right)=-\frac{1}{\left|S_j\right|}\sum _{x\in S_j}{\int }_{y\in R}p\left(y|x\right)log\left(p\left(y|x\right)\right)dy,}$ where $p\left(y|x\right)=N\left(x^T{\hat{a}}_j,{\hat{\sigma }}_j^2\right)$, |S_j| is the number of observations in node j. In Eq. (9), H(⋅) is differential Shannon entropy. After calculating the integral, we obtain ${\int }_{y\in R}p\left(y|x\right)log\left(p\left(y|x\right)\right)dy=\frac{1}{2}log\left(2\pi e{\hat{\sigma }}_j^2\right)$ (Cover & Thomas, 2006). Thus, $H\left(S_j\right)=\frac{1}{\left|S_j\right|}{\sum }_{x\in S_j}\frac{1}{2}log\left(2\pi e{\hat{\sigma }}_j^2\right)=\frac{1}{2}log\left(2\pi e{\hat{\sigma }}_j^2\right)$. Therefore, Eq. (9) can be rewritten as follows: (10)${\displaystyle I_j=\frac{1}{2}log\left(2\pi e\right)+log\left({\hat{\sigma }}_j\right)-\sum _{i\in \left\{L,R\right\}}\frac{\left|S_j^i\right|}{\left|S_j\right|}\left(\frac{1}{2}log\left(2\pi e\right)+log\left({\hat{\sigma }}_{j^i}\right)\right).}$

Second, let us introduce Renyi entropy-based information gain for node j: (11)${\displaystyle I_j=H_{\alpha }\left(S_j\right)-\frac{\left|S_j^L\right|}{\left|S_j\right|}{H}_{\alpha }\left(S_j^L\right)-\frac{\left|S_j^R\right|}{\left|S_j\right|}{H}_{\alpha }\left(S_j^R\right),H_{\alpha }\left(S_j\right)=\frac{1}{\left|S_j\right|}\sum _{x\in S_j}\frac{1}{1-\alpha }log\left({\int }_{y\in R}{p}^{\alpha }\left(y|x\right)dy\right),}$ where $p\left(y|x\right)=N\left(x^T{\hat{a}}_j,{\hat{\sigma }}_j^2\right)$, H_α(⋅) is differential Renyi entropy with parameter α; more precisely, it is a conditional Renyi entropy (Fehr & Berens, 2014). One can demonstrate that (12)${\displaystyle {\int }_{y\in R}{p}^{\alpha }\left(y|x\right)dy=\frac{1}{{\hat{\sigma }}^{\alpha -1}{\left(\sqrt{2\pi }\right)}^{\alpha -1}\sqrt{\alpha }}.}$ For example, this is shown in Nielsen and Nock (Nielsen & Nock, 2011). Thus, $H_{\alpha }\left(S_j\right)=log\left(\sqrt{2\pi }\right)+log\left({\hat{\sigma }}_j\right)-\frac{log\left(\sqrt{\alpha }\right)}{1-\alpha }$. Therefore, Eq. (11) can be rewritten in the following form: (13)${\displaystyle I_j^{\alpha }=log\left(\sqrt{2\pi }\right)+log\left({\hat{\sigma }}_j\right)-\frac{log\left(\sqrt{\alpha }\right)}{1-\alpha }-\sum _{i\in \left\{L,R\right\}}\frac{\left|S_j^i\right|}{\left|S_j\right|}\left(log\left(\sqrt{2\pi }\right)+log\left({\hat{\sigma }}_{j^i}\right)-\frac{log\left(\sqrt{\alpha }\right)}{1-\alpha }\right).}$

Third, Tsallis entropy-based information gain for node j can be formulated in the following way: (14)${\displaystyle I_j^{\beta }=H_{\beta }\left(S_j\right)-\frac{\left|S_j^L\right|}{\left|S_j\right|}{H}_{\beta }\left(S_j^L\right)-\frac{\left|S_j^R\right|}{\left|S_j\right|}{H}_{\beta }\left(S_j^R\right),H_{\beta }\left(S_j\right)=\frac{1}{\left|S_j\right|}\sum _{x\in S_j}\frac{1}{\beta -1}\left(1-{\int }_{y\in R}{p}^{\beta }\left(y|x\right)dy\right),}$ where $p\left(y|x\right)=N\left(x^T{\hat{a}}_j,{\hat{\sigma }}_j^2\right)$, H_β(⋅) is differential Tsallis entropy with parameter β. Using relation Eq. (12), we obtain that $H_{\beta }\left(S_j\right)=\frac{1}{\left|S_j\right|}{\sum }_{x\in S_j}\frac{1}{\beta -1}\left(1-\frac{1}{{\hat{\sigma }}_j^{\beta -1}{\left(\sqrt{2\pi }\right)}^{\beta -1}\sqrt{\beta }}\right)$. Therefore, Eq. (14) can be rewritten as: (15)${\displaystyle I_j^{\beta }=\frac{1}{\beta -1}\left(1-\frac{1}{{\hat{\sigma }}_j^{\beta -1}{\left(\sqrt{2\pi }\right)}^{\beta -1}\sqrt{\beta }}\right)-\sum _{i\in \left\{L,R\right\}}\frac{\left|S_j^i\right|}{\left|S_j\right|}\left(\frac{1}{\beta -1}\left(1-\frac{1}{{\hat{\sigma }}_{j^i}^{\beta -1}{\left(\sqrt{2\pi }\right)}^{\beta -1}\sqrt{\beta }}\right)\right).}$

Fourth, let us introduce Sharma-Mittal entropy-based information gain for node j: (16)${\displaystyle I_j=H_{\alpha ,\beta }\left(S_j\right)-\frac{\left|S_j^L\right|}{\left|S_j\right|}{H}_{\alpha ,\beta }\left(S_j^L\right)-\frac{\left|S_j^R\right|}{\left|S_j\right|}{H}_{\alpha ,\beta }\left(S_j^R\right),H_{\alpha ,\beta }\left(S_j\right)=\frac{1}{\left|S_j\right|}\sum _{x\in S_j}\frac{1}{1-\beta }\left({\left({\int }_{y\in R}{p}^{\alpha }\left(y|x\right)dy\right)}^{\frac{1-\beta }{1-\alpha }}-1\right),}$ where $p\left(y|x\right)=N\left(x^T{\hat{a}}_j,{\hat{\sigma }}_j^2\right)$, H_α,β(⋅) is differential Sharma-Mittal entropy with parameter α and β. Let us note that α, β > 0, α ≠ 1, β ≠ 1, α ≠ β according to the definition of Sharma-Mittal entropy. Using Eq. (12), we obtain that $H_{\alpha ,\beta }\left(S_j\right)=\frac{1}{\left|S_j\right|}{\sum }_{x\in S_j}\frac{1}{1-\beta }\left(\frac{1}{{\hat{\sigma }}_j^{\beta -1}{\left(\sqrt{2\pi }\right)}^{\beta -1}{\alpha }^{\frac{1-\beta }{2\left(1-\alpha \right)}}}-1\right)$. Therefore, Eq. (16) can be rewritten as follows: (17)${\displaystyle I_j^{\alpha ,\beta }=\frac{1}{1-\beta }\left(\frac{1}{{\hat{\sigma }}^{\beta -1}{\left(\sqrt{2\pi }\right)}^{\beta -1}{\alpha }^{\frac{1-\beta }{2\left(1-\alpha \right)}}}-1\right)-\sum _{i\in \left\{L,R\right\}}\frac{\left|S_j^i\right|}{\left|S_j\right|}\left(\frac{1}{1-\beta }\left(\frac{1}{{\hat{\sigma }}_{j^i}^{\beta -1}{\left(\sqrt{2\pi }\right)}^{\beta -1}{\alpha }^{\frac{1-\beta }{2\left(1-\alpha \right)}}}-1\right)\right).}$

In the framework of the introduced formulations of information gain, the best splitting corresponds to the maximum information gain value.

Random forest algorithm

Let us formulate the general algorithm of random forest construction considering the proposed expressions of information gain with different types of entropies. Suppose we have the following data (x₁, y₁), (x₂, y₂), .., (x_n, y_n) where each x_i represents a feature vector [x_i1, x_i2, …, x_ip] and let T be the number of trees we want to construct in our forest. Then, to build a random forest, the following steps should be performed:

• For t = 1, …, T:

  • Draw a bootstrap sample of size n from the data.

  • Grow a decision tree t from the bootstrapped sample by repeating the following steps until the minimum node size specified beforehand is reached, or the maximum depth of the tree is reached, or the minimum reduction in the information gain is reached:

    • sample $m=\sqrt{p}$ or m = [p/3] features (where p is the number of features in the dataset)

    • compute the information gain according to one of the Eqs. (5), (6), (7) for classification task and according to one of the Eqs. (10), (13), (15), (17) for regression task for each possible value among the bootstrapped data and m features

    • find the information gain maximum and split the node into two children nodes

• Output the ensemble of trees and combine the prediction of all trees into a single prediction according to Eq. (3) for a classification task and Eq. (4) for regression.