Constructing optimal binary decision trees is NP-complete☆

Using empirical models to predict whether sections within pipes have defects can save inspection costs and, potentially, avoid oil spills. Optimal Classification Tree (OCT) formulations offer potentially desirable combinations of interpretability and prediction accuracy on unseen pipes. Approaches based on powerful state-of-the-art OCT formulations have enabled researchers to solve decision tree problems optimally instead of using traditional sub-optimal greedy approaches. Yet, the recently proposed formulations also have limitations. Some of the most recent formulations require a large number of decision variables and constraints leading to computational inefficiencies. Previous formulations have optimal solutions with undesirable or invalid tree structures which may depend on the particular software implementation. Additionally, some formulations always grow a full tree even when desirable parsimonious tree options are available. This article proposes the Modified Optimal Classification Tree (M-OCT) formulation with novel leaf-branch-interaction constraints, which could stabilize the previous formulation and reduce the chance of invalid tree structures when generating optimal trees. By incorporating the idea of binary encoding of thresholds from a previous article, we reduce the total number of binary variables. We then extend M-OCT to construct a novel formulation called Binary Node Penalty Optimal Classification Tree (BNP-OCT) with binary splits and node complexity constraints, which support efficiency in standard branch-and-cut solvers and prevents the overfitting issue when learning the optimal tree models. We compare the proposed methods with alternatives including standard formulations using 15 standard data sets. In addition, we use 750 test cases to compare the computational stability of pre-existing formulations to those involving the proposed leaf-branch constraints. We demonstrate that the proposed formulation offers advantages in accuracy, computational efficiency, and structural stability. We also describe how the proposed methods are able to achieve 94% classification accuracy on balanced test sets for unseen pipes.

Decision trees are highly interpretable models for solving classification problems in machine learning (ML). The standard ML algorithms for training decision trees are fast but generate suboptimal trees in terms of accuracy. Other discrete optimization models in the literature address the optimality problem but only work well on relatively small datasets. Firat et al. (2020) proposed a column-generation-based heuristic approach for learning decision trees. This approach improves scalability and can work with large datasets. In this paper, we describe improvements to this column generation approach. First, we modify the subproblem model to significantly reduce the number of subproblems in multiclass classification instances. Next, we show that the data-dependent constraints in the master problem are implied, and use them as cutting planes. Furthermore, we describe a separation model to generate data points for which the linear programming relaxation solution violates their corresponding constraints. We conclude by presenting computational results that show that these modifications result in better scalability.

We present a novel regression algorithm that combines a Fuzzy Inference System (FIS) with a nature-inspired algorithm to predict variations in GroundWater Levels (GWLs). Initially, we considered several input features, including precipitation, temperature, evaporation, relative humidity, soil type, and GWL Lag. A feature importance analysis using regression tree ensemble learning reveals GWL lag as the most relevant feature and soil type as the least relevant. We eliminate features with low importance scores (soil type, temperature, and evaporation) to improve the computational efficiency. Our approach leverages Fuzzy C-Means (FCM) clustering to develop a Sugeno-type FIS with definite clusters. The dataset is clustered based on feature similarity, and Gaussian membership functions are assigned to each cluster. The mapping of each point is controlled by the range and standard deviation of the Gaussian membership functions. We train the model keeping 70% of the dataset, iteratively tuning the parameters of the Gaussian membership function for all clusters through the Invasive Weed Optimization (IWO) algorithm. The performance of the model is then evaluated using the remaining 30% of the datasets. The F-IWO-GWL model accurately predicts GWL fluctuations, achieving a high correlation coefficient (R = 0.89), low normalized root mean square error (nRMSE = 0.18), and minimal bias (bias = 0.08). A comparative analysis involving seventeen benchmark algorithms reveals the superior performance of our algorithm. To ensure a fair comparison, we calculated key metrics including Akaike’s Information Criterion (AIC), Bayesian Information Criterion (BIC), and Corrected AIC (AICc). The F-IWO-GWL algorithm exhibits the lowest values for AIC, BIC, and AICc among all tested algorithms, suggesting the best goodness-of-fit. This study provides a robust approach for predicting GWL fluctuation, applicable to various groundwater management scenarios. It offers valuable insights for policymakers and stakeholders involved in groundwater management, aiding informed decision-making.

The most efficient state-of-the-art methods to build classification trees are greedy heuristics (e.g. CART) that may fail to find underlying characteristics in datasets. Recently, exact linear formulations that have shown better accuracy were introduced. However they do not scale up to datasets with more than a few thousands data points. In this paper, we introduce four new formulations for building optimal trees. The first one is a quadratic model based on the well-known formulation of Bertsimas et al. We then propose two different linearizations of this new quadratic model. The last model is an extension to real-valued datasets of a flow-formulation limited to binary datasets (Aghaei et al.). Each model is introduced for both axis-aligned and oblique splits. We further prove that our new formulations have stronger continuous relaxations than existing models. Finally, we present computational results on 22 standard datasets with up to thousands of data points. Our exact models have reduced solution times with learning performances as strong or significantly better than state of the art exact approaches.

Decision trees (DTs) are a type of machine learning technique used for classification and regression problems. They are considered to be a part of explainable artificial intelligence due to their transparent and interpretable nature. The traditional approach in building DTs involves a top-down greedy search, which is usually fast but may result in a sub-optimal solution. An alternative method is evolutionary induction. It allows for more global exploration and can lead to simpler DTs, but it is computationally more expensive. To speed up the induction, different parallel approaches have been considered, with the GPU-based ones being the most efficient.

In this paper, we extend the GPU-supported algorithm of DT evolutionary induction to use less memory and time resources. An improved in-memory representation of DTs is introduced, which adapts depending on the tree structure. Two types are possible: compact and complete representations. A compact one is used for DTs that are sparsely filled. It contains only existing tree nodes and therefore the links between parent and child nodes must be explicitly stored (along with the data). On the other hand, a complete representation is applied when the tree levels are densely filled. Then, placeholders for all potential nodes are reserved and the parent position can be easily calculated based on the child’s place (and vice versa). The algorithm switches between the representations based on a density factor of tree filling.

To validate our concept, we conducted experiments on both real-life and artificial datasets with varying sizes and dimensions. The results show that the DT in-memory representation affects the time and memory requirements a lot. The most efficient is the adaptive one. It not only speeds up the induction time but also reduces memory requirements (in terms of both transfer and occupancy). As a result, the evolutionary induction of DTs is closer to being competitive with state-of-the-art greedy inducers in terms of computation time.

The interpretability of models has become a crucial issue in Machine Learning because of algorithmic decisions' growing impact on real-world applications. Tree ensemble methods, such as Random Forests or XgBoost, are powerful learning tools for classification tasks. However, while combining multiple trees may provide higher prediction quality than a single one, it sacrifices the interpretability property resulting in “black-box” models. In light of this, we aim to develop an interpretable representation of a tree-ensemble model that can provide valuable insights into its behavior. First, given a target tree-ensemble model, we develop a hierarchical visualization tool based on a heatmap representation of the forest's feature use, considering the frequency of a feature and the level at which it is selected as an indicator of importance. Next, we propose a mixed-integer linear programming (MILP) formulation for constructing a single optimal multivariate tree that accurately mimics the target model predictions. The goal is to provide an interpretable surrogate model based on oblique hyperplane splits, which uses only the most relevant features according to the defined forest's importance indicators. The MILP model includes a penalty on feature selection based on their frequency in the forest to further induce sparsity of the splits. The natural formulation has been strengthened to improve the computational performance of mixed-integer software. Computational experience is carried out on benchmark datasets from the UCI repository using a state-of-the-art off-the-shelf solver. Results show that the proposed model is effective in yielding a shallow interpretable tree approximating the tree-ensemble decision function.