Efficient Solving of Boundary Value Problems Using Radial Basis Function Networks Learned by Trust Region Method

A method using radial basis function networks (RBFNs) to solve boundary value problems of mathematical physics is presented in this paper. The main advantages of mesh-free methods based on RBFN are explained here. To learn RBFNs, the Trust Region Method (TRM) is proposed, which simplifies the process of network structure selection and reduces time expenses to adjust their parameters. Application of the proposed algorithm is illustrated by solving two-dimensional Poisson equation.


Introduction
Mesh-free methods of solving boundary value problems have been widely studied in different researches in the last decade [1].They belong to the class of projection methods [2][3][4][5][6].In this paper, we are studying one of the most promising mesh-free methods using the radial basis functions (RBFs).In the case of their application, the approximation of the decision is presented as the weighted sum of RBFs where weights are selected in such a way that the approximated solution satisfies the boundary value problem in the selected sample points.The major difficulty of using the mesh-free methods based on RBFs is the nonformalizable selection of basis function parameters.This issue can be overcome with radial basis function neural networks (RBFNs) [7][8][9][10][11].The process of boundary value problems' solution using RBFNs comes down to the RBFNs learning.The main difference between the RBFN method and other mesh-free methods is that it adjusts not only the weight of the basis functions but also their parameters [7][8][9][10][11].In [12], it was suggested to use the Trust Region Method (TRM) for training RBNFs [13].There are new results to improve the Trust Region Method performance in the submitted article, in particular the use of the Hessian matrix approximation to reduce the computation cost.Two approaches were considered for the initial allocation of radial basis function centers, and more optimal values of Trust Region Method hyperparameters were found.In Section 2, the RBFNs are discussed in order to solve the boundary value problems.Section 3 presents the proposed methodology of learning RBFNs using TRM.Section 4 presents the experimental analysis of the proposed solution to solve the boundary value problems.Finally, Section 5 presents the conclusion of the paper.
(ii) The second layer does the linear summation.RBF is used as a conversion function.
The output of a network is described by the expression where  is the quantity of RBFs,   is the weight of RBF, and p  is the RBF parameter vector .Gauss's functions, multiquadrics, and the reverse multiquadrics, among others, are examples of RBFs [14]. 2 International Journal of Mathematics and Mathematical Sciences Consider the process of solving boundary value problems using RBFNs using an example of the boundary value problem, given in the following operator form: where  is a desired solution;  is a differential operator;  is an operator of the boundary conditions; Ω is an area of solutions; Ω is a border of the area;  and  are known functions.The solution is approximated by RBFN: (1) Select  domestic and  boundary sample points from the sets Ω and Ω.
(2) Determine the structure of the network: the network type, the number of RBFs , and the RBFs type.
Specify initial values: w = ( 1 ,  2 , . . .,   ) are the weights of RBFs and p = (p 1 , p 2 , . . ., p  ) are parameters of RBFs (the structure of the vector p elements depends on the type of the functions; e.g., for a two-dimensional Gaussian function, it can be defined as p  = (  ,  1   ,  2  ), where  = 1 ⋅ ⋅ ⋅ ,   is the shape parameter (width), and ( 1  ,  2  ) is the position of its center).
(3) Learn the network; that is, find such values of w and p that the error functional representing the sum of squares of residuals in sample points reaches the minimum value: where is the selectable penalty factor, which takes into account the residuals in the boundary sample points.The trained network provides the solution at any arbitrary point when its coordinate comes to the input layer of the network.

Learning RBFN Using TRM
The efficiency of neural network method for solving boundary value problems depends on the efficiency of the method of solving the problem of the error functional minimization (3).TRM is one of the best approaches to solve problem (3) [12,15].The key features of this method are as follows: it has the possibility of simultaneous optimization of a large number of parameters; it has a high level of efficiency and convergence even for bad-conditioned problems; it can overcome local minimums; it has the capability of minimization of concave functions, that is, functions with negative definite Hessian matrix; and finally, when the second-order Taylor expansion is used as an approximating function, the problem of the objective function minimization reduces to the problem of minimizing a quadratic functional [13].All these make TRM ideal for solution of the problem of minimizing the error functional, which often has a large number of parameters to be optimized and many local minima, and it is an illconditioned problem.The basic idea of TRM is that, at each iteration  of the function (x) minimization, where x ∈ Ω ⊆   (Ω is the domain of definition and  is a set of real numbers), the function  is replaced by an approximating function   in the trust region   ⊆ Ω and the minimum of   is calculated in   , which becomes a new minimum of .
Depending on how the decrease predicted by the model is confirmed by the objective function, the decision on the expansion or contraction of the trust region is taken.There is a formal description of the algorithm.

Algorithm 1 (TRM).
Step 1 (initialization).Specify an initial value of x 0 , the radius of the trust region Δ 0 , the threshold accuracy of the estimated models  1 and  2 such that 0 <  1 ≤  2 < 1, the transformation coefficients  1 and  2 of the trust region (0 <  1 ≤  2 < 1), and the number of the iterations  = 0.
Step 2 (approximation of ).Select the norm ‖⋅‖ and construct a function   , approximating function  in the area   .
Step 3 (minimization of   ).Select the method of conditional minimization of   , which allows finding such a step s  that point x  + s  is a global minimum of   in   .
Step 4. This step consists in evaluation of the model accuracy   that is calculated as follows: If   ≥  1 , then x +1 = x  + s  ; otherwise, x +1 = x  .
Step 5. Change the trust region radius ( Step 6 (increasing the number of the iterations  =  + 1).If the required accuracy of the solution has been reached, or  is equal to the maximum number of iterations, or the radius of the trust region is too small, then complete the learning; otherwise, go back to Step 2.
Step 7 (stop).The algorithm is a generalized algorithm of TRM, as it has no instructions on how to construct a function Since the error functional is a twice differentiable function, hence the second-order Taylor series can be used as   model of ; as the norm, we are using the Euclidean norm here.To get the second-order Taylor series, it is necessary to calculate the Hessian matrix, which has a large computation cost.Instead of the exact value of the Hessian, we are using its approximate value, which is a multiplication of the Jacobi matrices H(x) ≈ G(x) = [J(x)]  J(x), where the Jacobian of the error function has the form where Using the second-order Taylor series leads to the need for solving the conditional problem of a quadratic functional minimization.To solve it, Steihaug's method [16] is used here.This method is a modification of the method of preconditioned conjugate directions, taking into account the restrictions on the solution (the solution should lie in   ) during   functional minimization and allowing working with negative definite Hessian matrix.

Experimental Study
As an example of solving boundary value problems using RBFN, learned by TRM, consider the boundary value problem for the two-dimensional Poisson equation, described in [8] This problem often arises in thermodynamics, electrostatics, and image processing.
In the second experiment, the RBFN consisted of 64 neurons, whose centers were located in the nodes of a uniform square grid bounded by points (−0.2; −0.2) and (1.2; 1.2); the neurons' width was constant and equal to 0.5.Initial values of weights of the RBFs were equal to zero.The solution error was equal to 5.3 ⋅ 10 −5 and was reached for 15 iterations.The error graph is shown in Figure 2.
As expected, the more the neurons in the network, the more accurately it approximates the desired solution.However, the process of learning such a network requires much more time (for learning the network from the first experiment, it needed two times fewer iterations than training a network from the second experiment).The dependency of the error  (10) on the iteration number  is shown in Figure 3.

Conclusion
In this paper, a method based on TRM was proposed for RBFNs learning.Approximate values of the Hessian matrix are used to improve its performance.This method allows reducing the time of network's learning.

Figure 3 :
Figure 3: The dependency of the error  (10) on the iteration number .