Numeric treatment of nonlinear second order multi-point boundary value problems using ANN, GAs and sequential quadratic programming technique

In this paper, computational intelligence technique are presented for solving multi-point nonlinear boundary value problems based on artificial neural networks, evolutionary computing approach, and active-set technique. The neural network is to provide convenient methods for obtaining useful model based on unsupervised error for the differential equations. The motivation for presenting this work comes actually from the aim of introducing a reliable framework that combines the powerful features of ANN optimized with soft computing frameworks to cope with such challenging system. The applicability and reliability of such methods have been monitored thoroughly for various boundary value problems arises in science, engineering and biotechnology as well. Comprehensive numerical experimentations have been performed to validate the accuracy, convergence, and robustness of the designed scheme. Comparative studies have also been made with available standard solution to analyze the correctness of the proposed scheme.


Introduction
Multi-point boundary value problems (BVPs) or non-local boundary value problems were introduced by Ilin and Moiseev (1987).These BVPs have become the most important area among researchers.These days, because of its widespread applications in engineering as to model the physical problems, including vibration happening in a wire of uniform cross section and combined of material with changed densities, through porous media applications, BVPs are used in fluid flow and in elastic stability.In addition, these problems have wide applications in applied physics particularly in heat conduction (Cannon, 1984;Cannon et al., 1987) and other problems in nonlinear elasticity (Timoshenko, 1961).By using perturbation techniques, these problems have been normally solved but these techniques have some confines, e.g. the approximate solution consists of a series of very small parameters, which poses difficulty since the popularity of nonlinear problems has no minor parameters.
These multi-point BVPs are generally limited to second order equation.Agarwal and Kinguradze (Das et al., 2010) solved higher order linear ordinary differential equation with singularity at numerous points in 2004.Cheung and Ren (Das et al., 2010) discussed multi-point BVPs in 2005.Tatari and Dehghan (Das et al., 2010) used Adomian decomposition method (ADM) to find the solution of multipoint BVP in 2006.Gupta (1992Gupta ( , 1994Gupta ( , 1997) ) is believed to be the first who used functional analytical technique to demonstrate the existence of the solution monitored by Eloe and Henderson (1994), Ma (1999Ma ( , 2001)), Liu (2002), Webb (2001), etc. Deimling, (1985), Guo and Lakshmikantham (1988) and Krasnosel'skij (1964) used commonly fixed-point theorem on cones for positive solution.The source to apply this theorem can be initiated with semi linear elliptic equations on annuli (Erbe & Wang, 1994).
Stochastic numerical techniques have been used extensively by many researchers in different fields to solve linear and nonlinear differential equations (Raja & Samar, 2012;Khan, 2009).Interest in this area is significantly enhanced with effective numerical treatment for fractional differential equation (Khan et al., 2012).The analysis is carried out further by constructing different types of neural networks models with/without satisfying the boundary conditions, exactly.Furthermore, the effect of change on number on neurons on accuracy and convergence has also been analyzed.
In the present study, computational intelligent techniques are developed for solving a nonlinear multipoint BVPs representing to model the equation with the help of universal approximation capability of feed-forward artificial neural networks (ANN) trained in an unsupervised manner with genetic algorithms (GA) technique, as a tool for effective global search optimizer, hybrid with sequential quadratic programming technique (SQPT), an efficient local search approach.The proposed schemes are evaluated on four variants of the problems to validate the correctness of the approaches.The reliability and effectiveness of the proposed scheme is analyzed based on sufficient large set of data generated from independent execution of the optimizers, i.e., GA, SQPT and GA-SQPT.
The organization of the paper is as follows: in the second section, the system model of the problem is given in details.Design methodology consists of feed-forward artificial neural networks modeling and learning procedure based on GA, SQPT and GA-SQPT is narrated in section three.Numerical experimentation of number of variant of the problem is provided in section four.Concluding remarks and few future research directions are provided in the last section.

General form of Multi-point BVPs
Mathematically, this problem is a second-order ordinary differential equation (ODE).The generic form or general form of multi-point boundary value problems is written as follows, The Eq. ( 1) represents a standard form of multi-point BVP.The aim of studying the generalized form of the nonlinear multi-point BVP and its solutions is to develop a broader understanding of the general equations and to develop analytical tools to analyze these equations to cover more applications as they arise in the future.

Proposed Methodology
The brief description of designed methodology will be presented for the solution of the multi-point BVPs.In this section, the procedure consists of the development of two feed-forward unsupervised neural networks models of the equation.Moreover, brief introductory material for the optimization algorithms i.e. evolutionary computing and active set programming techniques are also given.

Neural Network Mathematical Modeling
Mathematical model for the second order multi-point boundary value problems is constructed with the strength of feed-forward ANN, in the form of following continuous mapping for the solution y(x) , and its first dy/dx, second d 2 y/dx 2 respectively, For the activation function log-sigmoid function is used in the neural network modeling, given as: where the vectors   and , w are real-valued bounded unknown parameters, i.e., weights W = [   , , w ], n is the number of neurons and g is the activation function generally taken as log-sigmoid function.
The suitable combination of the networks given in the set (2) can be used to model the differential equation ( 1).An objective function or fitness function is developed in an unsupervised manner and it is defined by sum of two mean square error, e 1 is error function associated with differential equation (1) and it is given as: Similarly, e 2 is the error function associated with boundary conditions for Eq. ( 1) is given as: A second form of feed-forward artificial neural networks modeling is also constructed such that initial and boundary conditions are exactly satisfied in order to solve boundary value problems of the multipoint BVPs.The governing equations of alternative form of artificial neural networks model for the solution y(x), and its first dy/dx, second d 2 y/dx 2 respectively, written as: where ŷ(x), dŷ/dx, d 2 ŷ/dx 2 are the networks as shown in Eq. ( 6).The updated neural network can be developed for solving the general nonlinear multi-point boundary value problem (1) that satisfies boundary conditions exactly by taking the approximate values of P(x) and Q(x).
An objective function or fitness function is developed in an unsupervised manner for introducing error function base on the sum of mean square error as: One can only deduce that with the availability of weights of neural networks such that the objective function given in Eq. ( 3) or Eq. ( 6) approaches zero, the proposed solution will approximate the exact solution of the problem closely.

Learning Methodology
The procedure for training the designed parameters of the networks is accomplished with the help of genetic algorithms (GAs), sequential quadratic programming technique (SQPT) and hybrid computing approach GA-SQPT.The recent researches in genetic algorithms (GAs) has emphasized that the initial proposals were found incompetent of solving tough problems in a valuable and efficient way.For a large numbers of optimization problems, the time of execution of first-generation GAs increases significantly while solution quality decreases.Moreover the things such as selection schemes, encoding procedures, and self-adaptive and knowledge based operators play an important role in the optimization of well convex and stochastic nature problems.Since the beginning of the GAs, it has used for optimization of different fields of interest.In addition to the optimization problems, it is also used as an effective technique for achieving modeling, forecasting, control and simulation objectives.Traditional optimization approach start with a single candidate and search iteratively for the optimal result by using static heuristics.While, the GA techniques uses a population of candidate to seek a number of areas of a solution space, simultaneously and adaptively.The most famous methods that go beyond easy local search are GAs, simulated annealing (SA), and tabu search (TS) etc. Genetic algorithms work on a population of individuals.All individual is a potential solution to a given problem and is normally encoded as a fixed-length binary string, which is an analogy with an actual chromosome.After an initial population is randomly or heuristically generated, the algorithm evolves population through sequential and iterative application of selection, crossover, and mutation operators.A new generation is shaped after every each iteration.The strongest feature of the GA is that do not get stuck in local minimum.
Sequential quadratic programming techniques is basically a local search method which belongs to the class of efficient constrained optimization technique.The method is based on mathematical model for solving the Karush-Kuhn-Tucker (KKT) equations to get the desired Lagrange multipliers simply and consequently the global candidate solution.The SQP technique has been used in many practical important optimization problems since their origin.The interesting reader is referred to these literatures to known the history, importance, subject term, working and detail procedural step of the algorithm.The necessary procedure adopted for GA-SQPT is given in following steps: Step 1: Initialization: Initial values of parameters are set in this step with random assignment and declarations.These setting are also tabulated in Table1 for important parameter of GAs.
Step 2: Fitness Evaluation: Calculate the fitness of each individual or chromosome of population using the Eq. ( 2) and Eq. ( 5), for first and second type of modeling, respectively.
Step 3: Termination Criteria: Terminate the algorithm when either of following criteria matches:  Predefined fitness values e  10 -15 is achieved.
 Predefine number of generations are executed.
 Any of termination setting given in Table 1 for GA is fulfilled.
If termination criterion meets, then go to step 5 Step 4: Reproduction: Create next generation on the basis of Crossover: Call for scattered function, Mutation: Call for Adaptive Feasible function, Selection: Call for Stochastic Uniform function and Elitism account is set 4 etc.
Repeat the procedure from step 2 to step 4 with newly produced population Step 5: Improvements: Sequential quadratic programming technique is used for further refinement of results by taking final adaptive weights of GAs as initial weights (start point) of SQPT algorithm.
SQPT is applied as per setting of parameters given in Table 2. Store also the refined final weights of the algorithm.
Step 6: Neurons Analysis: Repeat step 1 to 5 for by taking size of initial weights, i.e. 15, 30 and 45 for k = 5, 10 and 15 neurons, respectively.These results are used for detail analysis of algorithm later.
The parameter setting used in this paper for GA algorithm in table 1 and SQPT algorithm are given in Table 2.

Simulations and Results
In this section, three problems of multi-point boundary value problems will be presented with their multi-point boundary conditions.We compare the results of all the numerical problems for different number of neurons with exact solutions that will be provided with each problem.The error analysis showed graphically the high reliability of purposed scheme.
The exact solution of the problem is given as: The proposed methodologies are applied to solve the problem by taking 10 neurons in the hidden layer of neural networks thus with a total 30 design parameter or weights, ( , , ) W w   .The fitness functions e and ε is formulated for case for r ϵ (0, 1] with a step size h = 0.1 as: of the Eq. ( 7) is obtained using the optimal weights of algorithms in first equation of the set (2) and ( 5), respectively.The results due to GA-SQPT for the case k = 5 are given in simplify form as: To find the solution of the problems, we have applied the GA, SQPT and hybrid approach GA-SQPT by invoking the Matlab built in functions using the parameters setting as given in Table 1 and Table 2.
The results calculated for the both neural networks models optimized with GA-SQPT for different number of neurons, i.e., k = 5, 10 and 15, are given in the Table 3 for inputs t between 0 and 1 with a step size of 0.1.The results obtained with exact solution given in Eq. ( 8) are also given in the table for the same inputs.It is seen that the proposed solution match with exact solution to five to seven decimal space of the accuracy.The values of absolute error (AE) are determined for each algorithm, i.e., GA, SQPT and GA-SQPT, for both models and results are shown graphically in Fig 2  for SQPT lies in the range 10 -05 to 10 -07 , 10 -05 to 10 -06 , and 10 -05 to 10 -06 for k = 5, 10 and 15 respectively using first fitness function, while for second fitness function these values lies in the range 10 -06 to 10 -07 10 -06 to 10 -07 , and 10 -07 to 10 -08 for k = 5, 10 and 15, respectively.Whereas, from Fig 3(c) the values of AE for GA-SQPT lies in the range 10 -02 to 10 -05 , 10 -02 to 10 -03 , and 10 -03 to 10 -05 for k = 5, 10 and 15 respectively using first fitness function, while for second fitness function these values lies in the range 10 -06 to 10 -08 , 10 -05 to 10 -07 , and 10 -06 to 10 -07 for k = 5, 10 and 15, respectively.

250772371
] 1 e  (16) The results obtained for the both neural networks models of GA-SQPT for k = 5, 10 and 15, are also presented in the Table 4 along with exact solution using Eq. ( 13).
The exact solution of the problem is given as: The proposed methodologies are applied to solve the problem by taking 10 neurons in the hidden layer of neural networks thus with a total 30 design parameter or weights, W (   , , w ).The fitness functions e and ε is formulated for this case for r ϵ (0, 1] with a step size h = 0.1 as: The proposed solution ) ( ˆx y and ) ( ~x y of the Eq. ( 17) is obtained using the optimal weights of algorithms in first equation of the set (2) and ( 5), respectively.The results due to GA-SQP for the case k = 15 are given in simplify form as: The results obtained for the both neural networks models of GA-SQPT for k = 5, 10 and 15, are also presented in the Table 4 along with exact solution using Eq. ( 18).The proposed solution approximately matches with exact solution up to four to six decimal places of the accuracy.
 Better hardware platform like parallel, grid and cloud computing, can help in decreasing computational time of the algorithm, considerably.Consequently, we may look for other complicated nonlinear problems to solve by our design scheme.

Fig. 1 .
Fig. 1.Set of optimal weights along with parameters for problem 1 by GA-SQPT using 5, 10 and 15 numbers of neurons The proposed solution ) ( ˆx y and ) ( ~x yof the Eq.(7) is obtained using the optimal weights of algorithms in first equation of the set (2) and (5), respectively.The results due to GA-SQPT for the case k = 5 are given in simplify form as:

Fig. 2 .
Fig. 2. Results based on AE for two models in case of problem I (solid and dotted lines for neural network with/without satisfying boundary condition exactly) (a) for GA (b) for SQP and (c) for GA-SQP It is seen that the values of AE are determined for each algorithm, i.e., GA, SQPT and GA-SQPT, for both models and results are shown graphically in Fig 3.In 3(a), the values of AE for GA lies in the range 10 -05 to 10 -06 , 10 -05 to 10 -06 , and 10 -05 to 10 -06 for k=5, 10 and 15 respectively using first fitness function, while for fitness second function these values lies in the range 10 -07 to 10 -08 , 10 -06 to 10 -07 , and 10 -05 to 10 -06 for k = 5, 10 and 15, respectively.Similarly it is inferred from Fig 3(b) the value of AE for SQPT lies in the range 10 -05 to 10 -07 , 10 -05 to 10 -06 , and 10 -05 to 10 -06 for k = 5, 10 and 15 respectively using first fitness function, while for second fitness function these values lies in the range 10 -06 to 10 -07 10 -06 to 10 -07 , and 10 -07 to 10 -08 for k = 5, 10 and 15, respectively.Whereas, from Fig3(c) the values of AE for GA-SQPT lies in the range 10 -02 to 10 -05 , 10 -02 to 10 -03 , and 10 -03 to 10 -05 for k = 5, 10 and 15 respectively using first fitness function, while for second fitness function these values lies in the range 10 -06 to 10 -08 , 10 -05 to 10 -07 , and 10 -06 to 10 -07 for k = 5, 10 and 15, respectively.
absolute error AE are determined for each algorithm, i.e., GA, SQPT and GA-SQPT for both models and results are shown graphically in Fig 4.

Fig. 4 .
Fig. 4. Results based on AE for two models in case of problem II (solid and dotted lines for neural network with/without satisfying boundary condition exactly) (a) for GA (b) for SQPT and (c) for GA-SQPT