Discrete optimization methods to fit piecewise affine models to data points

Highlights

• We address the fundamental piecewise affine model fitting problem.

• We propose an MILP formulation, strengthened with symmetry breaking constraints.

• We present a heuristic with point-reassignment and domain partition at each iteration.

• We report extensive results on real-world and structured random instances.

• We conclude with an application to the identification of hybrid dynamical systems.

Abstract

Fitting piecewise affine models to data points is a pervasive task in many scientific disciplines. In this work, we address the k-Piecewise Affine Model Fitting with Piecewise Linear Separability problem (k-PAMF-PLS) where, given a set of m points $\{{\mathbf{a}}_1,\dots ,{\mathbf{a}}_m\}\subset {\mathbb{R}}^n$ and the corresponding observations $\{b_1,\dots ,b_m\}\subset \mathbb{R}$, we have to partition the domain ${\mathbb{R}}^n$ into k piecewise linearly (or affinely) separable subdomains and to determine an affine submodel (i.e., an affine function) for each of them so as to minimize the total linear fitting error w.r.t. the observations b_i.

To solve k-PAMF-PLS to optimality, we propose a Mixed-Integer Linear Programming (MILP) formulation where symmetries are broken by separating shifted column inequalities. For medium-to-large scale instances, we develop a four-step heuristic involving, at each iteration, a point reassignment step based on the identification of critical points and a domain partition step based on multicategory linear classification. Differently from traditional approaches proposed in the literature for similar fitting problems, in both our exact and heuristic methods the domain partitioning and submodel fitting aspects are taken into account simultaneously.

Computational experiments on real-world and structured randomly generated instances show that, with our MILP formulation with symmetry breaking constraints, we can solve to proven optimality many small-size instances. Our four-step heuristic turns out to provide close-to-optimal solutions for the small-size instances, while allowing to tackle instances of much larger size. The experiments also show that the combined impact of the main features of our heuristic is quite substantial when compared to standard variants not including them.

We conclude the paper with an application to the identification of dynamical piecewise affine systems, for which we obtain promising results of comparable quality with those achieved with state-of-the-art methods from the literature (hinging hyperplane models and sum-of-norms regularization) on benchmark data sets.

Introduction

Fitting a set of data points in ${\mathbb{R}}^n$ with a combination of low complexity models is a pervasive problem in, essentially, any area of science and engineering. It naturally arises, for instance, in prediction and forecasting when determining a model to approximate the value of an unknown function, or whenever one wishes to approximate a highly complex nonlinear function with a simpler one. Applications range from optimization (see, e.g., [36] and the references therein) to statistics (see, e.g., the recent work in [9]), to data mining (see, e.g., [5], [11]), and to system identification (see, for instance, [18], [6], [35]), only to cite a few.

Among the different options, piecewise affine models have a number of advantages with respect to other model fitting approaches. Indeed, they are compact and simple to evaluate, visualize, and interpret, in contrast to models obtained with other techniques such as, e.g., neural networks, while allowing to approximate even highly nonlinear functions.

Given a set of m points $\mathcal{A}=\{{\mathit{a}}_1,\dots ,{\mathit{a}}_m\}\subset {\mathbb{R}}^n$, with index set $I=\{1,\dots ,m\}$, the corresponding observations $\{b_1,\dots ,b_m\}\subset \mathbb{R}$, and a positive integer k, the general problem of fitting a piecewise affine model to the data points $\left\{\right({\mathit{a}}_1,b_1),\dots ,({\mathit{a}}_m,b_m\left)\right\}$ consists in partitioning the domain ${\mathbb{R}}^n$ into k continuous subdomains $D_1,\dots ,D_k$, with index set $J=\{1,\dots ,k\}$, and in determining, for each subdomain D_j, an affine submodel (an affine function) $f_j:D_j\to \mathbb{R}$, so as to minimize a measure of the total fitting error. Adopting the notation $f_j\left(\mathit{x}\right)={\mathit{w}}^j\mathit{x}-w_0^j$ with coefficients $({\mathit{w}}^j,w_0^j)\in {\mathbb{R}}^{n+1}$, the j-th affine submodel corresponds to the hyperplane $H_j=\left\{\right(\mathit{x},f_j\left(\mathit{x}\right))\in {\mathbb{R}}^{n+1}:f_j(\mathit{x})={\mathit{w}}^j\mathit{x}-w_0^j\}$, defined for all $\mathit{x}\in D_j$.¹ The total fitting error is defined as the sum, over all $i\in I$, of a function of the difference between b_i and the value $f_{j\left(i\right)}\left({\mathit{a}}_i\right)$ provided by the piecewise affine model, where j(i) is the index of the affine submodel corresponding to the subdomain $D_{j\left(i\right)}$ which contains the point ${\mathit{a}}_i$.

In the literature, different error functions (e.g., linear or quadratic) as well as different types of domain partition (with linearly or nonlinearly separable subdomains) have been considered. See Fig. 1(a) for an illustration of the case with k=2 and a domain partition with linearly separable subdomains.

In this work, the focus is on the version of the general piecewise affine model fitting problem with a linear error function (ℓ₁ norm) and a domain partition with piecewise linearly separable subdomains.² We refer to it as to the k-Piecewise Affine Model Fitting with Piecewise Linear Separability problem (k-PAMF-PLS). A more formal definition of the problem will be provided in Section 3.

k-PAMF-PLS shares a connection with the so-called k-Hyperplane Clustering problem (k-HC), an extension of a classical clustering problem which calls for k hyperplanes in ${\mathbb{R}}^{n+1}$ which minimize the sum, over all the data points $\left\{\right({\mathit{a}}_1,b_1),\dots ,({\mathit{a}}_m,b_m\left)\right\}$, of the ℓ₂ distance from $({\mathit{a}}_i,b_i)$ to the hyperplane the point is assigned to. See [8], [1], [13], [14] for some recent work on the problem and [4] for the problem variant aiming at minimizing the number of hyperplanes needed to fit all the points within a prescribed tolerance $\epsilon >0$.

It is nevertheless crucial to note that, differently from many of the approaches in the literature (which we briefly summarize in Section 2) and depending on the type of the domain partition that is adopted, a piecewise affine function cannot be determined by just solving an instance of k-HC. A naive application of algorithms designed for k-HC to tackle k-PAMF-PLS can indeed lead to solutions with a large fitting error, as a consequence of the domain partitioning aspect being entirely neglected. As illustrated in Fig. 1(b), the two aspects of k-PAMF-PLS, namely, submodel fitting and domain partitioning, should be taken into account at once to obtain a solution where the two are consistent. For this reason, most of the algorithms in the literature incorporate techniques to enforce the piecewise linear separability of the domain, which is typically not guaranteed in solutions of k-HC and its variants. In this work, we propose exact methods for k-PAMF-PLS based on mixed-integer linear programming as well as heuristic algorithms which consider both aspects of the problem simultaneously, rather than deferring the domain partitioning aspect to a later stage of the solution process.

The paper is organized as follows. After summarizing previous and related works in Section 2, we formally define the problem under consideration in Section 3. In Section 4, we provide a Mixed-Integer Linear Programming (MILP) formulation for k-PAMF-PLS. We then strengthen the formulation when using it for solving the problem in a branch-and-cut setting by generating symmetry-breaking constraints. In Section 5, we propose a four-step heuristic to tackle larger-size instances. Computational results are reported and discussed in Section 6. In Section 7, we consider an application of k-PAMF-PLS to problems in the area of dynamical system identification and compare the obtained results with those provided by state-of-the-art methods. Section 8 contains some concluding remarks. Portions of this work appeared, in a preliminary stage, in [2], [3].