Variable Fidelity Regression Using Low Fidelity Function Blackbox and Sparsification

Abstract

We consider construction of surrogate models based on variable fidelity samples generated by a high fidelity function (an exact representation of some physical phenomenon) and by a low fidelity function (a coarse approximation of the exact representation). A surrogate model is constructed to replace the computationally expensive high fidelity function. For such tasks Gaussian processes are generally used. However, if the sample size reaches a few thousands points, a direct application of Gaussian process regression becomes impractical due to high computational costs. We propose two approaches to circumvent this difficulty. The first approach uses approximation of sample covariance matrices based on the Nyström method. The second approach relies on the fact that engineers often can evaluate a low fidelity function on the fly at any point using some blackbox; thus each time calculating prediction of a high fidelity function at some point, we can update the surrogate model with the low fidelity function value at this point. So, we avoid issues related to the inversion of large covariance matrices — as we can construct model using only a moderate low fidelity sample size. We applied developed methods to a real problem, dealing with an optimization of the shape of a rotating disk.

Appendices

Appendix

A Proof of Technical Statements

In this section we provide the proofs of the statements of Sect. 4.

Proof (Proof of Statement 1)

For the posterior mean we get:

$$\begin{aligned} \hat{\mathbf {y}}_h(\mathbf {x}^*)&\approx \mathbf {K}_1^* \mathbf {K}_{11}^{-1} \mathbf {K}_1^T (\mathbf {K}_1 \mathbf {K}_{11}^{-1} \mathbf {K}_1^T + \mathbf {R}^{-2})^{-1} \mathbf {y}= \mathbf {K}_1^* \mathbf {K}_{11}^{-1} \mathbf {K}_1^T \mathbf {R}(\mathbf {R}\mathbf {K}_1 \mathbf {K}_{11}^{-1} \mathbf {K}_1^T \mathbf {R}+ \mathbf {I}_{n})^{-1} \mathbf {R}\mathbf {y}=\\&= \mathbf {K}_1^* \mathbf {K}_{11}^{-1} \mathbf {C}_1^T (\mathbf {C}_1 \mathbf {K}_{11}^{-1} \mathbf {C}_1^T + \mathbf {I}_{n})^{-1} \mathbf {R}\mathbf {y}= \mathbf {K}_1^* \mathbf {K}_{11}^{-1} (\mathbf {C}_1^T \mathbf {C}_1 \mathbf {K}_{11}^{-1} + \mathbf {I}_{n_1})^{-1} \mathbf {C}_1^T \mathbf {R}\mathbf {y}= \\&= \mathbf {K}_1^* (\mathbf {C}_1^T \mathbf {C}_1 + \mathbf {K}_{11})^{-1} \mathbf {C}_1^T \mathbf {R}\mathbf {y}= \mathbf {K}_1^* (\mathbf {C}_1^T \mathbf {C}_1 + \mathbf {V}^T_{11} \mathbf {V}_{11})^{-1} \mathbf {C}_1^T \mathbf {R}\mathbf {y}=\\&= \mathbf {K}_1^* \mathbf {V}^{-1}_{11} (\mathbf {V}^{-T}_{11} \mathbf {C}_1^T \mathbf {C}_1 \mathbf {V}_{11}^{-1} + \mathbf {I}_{n_1})^{-1} \mathbf {V}^{-T}_{11} \mathbf {C}_1^T \mathbf {R}\mathbf {y}= \mathbf {K}_1^* \mathbf {V}^{-1}_{11} (\mathbf {I}_{n_1} + \mathbf {V}^T \mathbf {V})^{-1} \mathbf {V}^T \mathbf {R}\mathbf {y}. \end{aligned}$$

We use the same approach to derive an equation for the posterior variance:

$$\begin{aligned} \mathbb {V} \left( X^* \right) - (\rho ^2 \sigma _l^2 + \sigma _d^2) \mathbf {I}_{n^*}&\approx \mathbf {K}_1^* \mathbf {K}_{11}^{-1} \mathbf {K}_1^{*T} - \mathbf {K}_1^* \mathbf {K}_{11}^{-1} \mathbf {K}_1^T (\mathbf {R}^{-2} + \mathbf {K}_1 \mathbf {K}_{11}^{-1} \mathbf {K}_1^T)^{-1} \mathbf {K}_1 \mathbf {K}_{11}^{-1} \mathbf {K}_1^{*T}=\\&= \mathbf {K}_1^* (\mathbf {K}_{11}^{-1} - \mathbf {K}_{11}^{-1} \mathbf {K}_1^T (\mathbf {R}^{-2} + \mathbf {K}_1 \mathbf {K}_{11}^{-1} \mathbf {K}_1^T)^{-1} \mathbf {K}_1 \mathbf {K}_{11}^{-1}) \mathbf {K}_1^{*T}=\\&= \mathbf {K}_1^* (\mathbf {K}_{11} + \mathbf {K}_1^T \mathbf {R}^2 \mathbf {K}_1)^{-1} \mathbf {K}_1^{*T} = \mathbf {K}_1^* (\mathbf {V}^T_{11} \mathbf {V}_{11} + \mathbf {C}_1^T \mathbf {C}_1)^{-1} \mathbf {K}_1^{*T}= \\&= \mathbf {K}_1^* \mathbf {V}^{-1}_{11} (\mathbf {I}_{n_1} + \mathbf {V}^T \mathbf {V})^{-1} \mathbf {V}^{-T}_{11} \mathbf {K}_1^{*T}. \end{aligned}$$

Proof (Proof of Statement 2)

First of all we have to calculate the matrices $\mathbf {V}_{11}$ and $\mathbf {V}= \mathbf {R}\mathbf {K}_1 \mathbf {V}_{11}^{-T}$. The matrix $\mathbf {V}_{11}$ is of size $n_1 \times n_1$, so we need $O(n_1^3)$ to get its inverse. To calculate $\mathbf {K}_1 \mathbf {V}_{11}^{-T}$ we need $O(n_1^2 n)$ operations. Finally, as $\mathbf {R}$ is a diagonal matrix, we use $O(n_1 n)$ operations to get $\mathbf {V}$.

In case $n^* = 1$ to get the posterior mean we have to calculate $\mathbf {V}_{11} (\mathbf {I}_{n_1} + \mathbf {V}^T \mathbf {V})^{-1} \mathbf {V}^T \mathbf {y}$. We use $O(n_1^2 n)$ operations to calculate $\mathbf {V}^T \mathbf {V}$, to inverse $\mathbf {I}_{n_1} + \mathbf {V}^T \mathbf {V}$ we need $O(n_1^3)$ operations, to calculate $\mathbf {V}_{11} (\mathbf {I}_{n_1} + \mathbf {V}^T \mathbf {V})^{-1} \mathbf {V}^T$ one uses extra $O(n_1^2 n)$ operations, and finally to calculate the posterior mean we need additional $O(n_1 n)$ operations. Consequently, to calculate the posterior mean we use $O(n_1^2 n)$ operations.

In the same way in order to calculate $\mathbf {V}_{11} (\mathbf {I}_{n_1} + \mathbf {V}^T \mathbf {V})^{-1} \mathbf {V}_{11}^{-1}$ we need $O(n_1^2 n)$ operations to calculate $(\mathbf {I}_{n_1} + \mathbf {V}^T \mathbf {V})^{-1}$ and additional $O(n_1^3)$ operations to get the final matrix. Consequently, in order to calculate the posterior variance we use $O(n_1^2 n)$ operations.

Finally, we need $O(n_1^2 n)$ operations to compute the required matrices, and $O(n_1^2 n)$, to obtain the posterior mean and the posterior variance from these precomputed matrices. So, the total computational complexity is $O(n_1^2 n)$.

B Comparison of Low and High Fidelity Model for Rotating Disk

There are two available solvers for $u_\mathrm{max}$ and $s_\mathrm{max}$ calculation. The low fidelity function is calculated using Ordinary Differential Equations (ODE) solver based on a simple Runge–Kutta’s method. The high fidelity function is calculated using Finite Element Model (FEM) solver from ANSYS.

To compare the solvers we draw the scatter plots of low and high fidelity values and also plot slices of the corresponding functions. We generate a random sample of points in a specified design space box, calculate the low and high fidelity function values and draw the low fidelity function values versus the high fidelity function values at the same points. The scatter plots are in Fig. 3: the difference between values increases significantly when the values are increasing.

Fig. 3.

Comparison of the high and the low fidelity solvers via scatter plots

For the central point of the design space box with $r_1 = 0.06, r_2 = 0.13, r_3 = 0.16, r_4 = 0.185, t_1 = 0.027, t_3 = 0.027$ we construct one-dimensional slices by varying single input variable in specified bounds. Slices for different input variables for $u_\mathrm{max}$ and for $s_\mathrm{max}$ are given in Fig. 4. In case of $u_\mathrm{max}$ the high and the low fidelity functions have the same behaviour, and the low fidelity function models the high fidelity function accurately. For $s_\mathrm{max}$ the high and the low fidelity functions are sometimes different: their behaviours differ for a slice along $r_1$ input, and local maxima differ for slice along $t_3$ input.

Fig. 4.

Comparison of the high and the low fidelity solvers via outputs’ slices