Design Feed Forward Neural Network To Solve Boundary Value Problems

: The aim of this paper is to design fast feed forward neural network to present a method to solve second order boundary value problem for ordinary differential equations. That is to develop an algorithm which can speedup the solution times, reduce solver failures, and increase possibility of obtaining the globally optimal solution and we use several different training algorithms many of them having a very fast convergence rate for reasonable size networks.Finally, we illustrate the method by solving model problem and present comparison with solutions obtained using other different method .


Introduction
Many methods have been developed so far for solving differential equations.Some of them produce a solution in the form of an array that contains the value of the solution at a selected group of points, others use basis functions to represent the solution in analytic form and transform the original problem usually to a system of algebraic equations.[1] Most of the previous study in solving differential equations using Artificial neural network(ANN) is restricted to the case of solving the systems of algebraic equations which result from the discretization of the domain.ANN is a simplified mathematical model of the human brain, It can be implemented by both electric elements and computer software.It is a parallel distributed processor with large numbers of connections, it is an information processing system that has certain performance characters in common with biological neural networks.Ann have been developed as generalizations of mathematical models of human cognition or neural biology, based on the assumptions that : [1] 1-Information processing occurs at many simple elements called neurons that is fundamental to the operation of ANN's.2-Signals are passed between neurons over connection links.3-Each connection link has an associated weight which, in a typical neural net, multiplies the signal transmitted.4-Each neuron applies an activation function (usually nonlinear) to its net input (sum of weighted input signals) to determine its output signal.
The units in a network are organized into a given topology by a set of connections or weights .
ANN is Characterized by [2] : 1-Architecture: its pattern of connections between the neurons.2-Training Algorithm : its method of determining the weights on the connections.3-Activation function.
ANN are often classified as single layer or multilayer.In determining the number of layers, the input units are not counted as a layer, because they perform no computation.Equivalently, the number of layers in the net can be defined to be the number of layers of weighted interconnects links between the slabs of neurons [3].

2.
Multilayer Feed Forward Architecture [4] In a layered neural network the neurons are organized in the form of layers.We have at least two layers: an input and an output layer.The layers between the input and the output layer (if any) are called hidden layers, whose computation nodes are correspondingly called hidden neurons or hidden units.Extra hidden neurons raise the network's ability to extract higher-order statistics from (input) data .
The ANN is said to be fully connected in the sense that every node in each layer of the network is connected to every other node in the adjacent forward layer , otherwise the network is called partially connected.Each layer consists of a certain number of neurons; each neuron is connected to other neurons of the previous layer through adaptable synaptic weights w and biases b .
Print to PDF without this message by purchasing novaPDF (http://www.novapdf.com/)

Description of the Method
In the proposed approach the model function is expressed as the sum of two terms: the first term satisfies the boundary conditions (BC) and contains no adjustable parameters.The second term can be found by using feed forward neural network(FFNN) which is trained so as to satisfy the differential equation and such technique we called collocation neural network.Since it is known that a multilayer FFNN with one hidden layer can approximate any function to arbitrary accuracy [5], [6] , thus our FFNN contains one hidden layer.In this section we will illustrate how our approach can be used to find the approximate solution of the general form a differential equation of 2nd order : y"(x) = F( x, y(x), y'(x) ) , (1) where a subject to certain BC's and x  (x1, x2, …, xn)  Rn, D  Rn denotes the domain and y(x) is the solution to be computed.
If yt(x, p) denotes a trial solution with adjustable parameters p, the problem is transformed to a discretize form : subject to the constraints imposed by the BC's.In the our proposed approach, the trial solution yt employs a FFNN and the parameters p correspond to the weights and biases of the neural architecture.We choose a form for the trial function yt(x) such that it satisfies the BC's.This is achieved by writing it as a sum of two terms : yt(xi , p) = A(x) + G( x, N(x, p) ) , (3) where N(x, p) is a single-output FFNN with parameters p and n input units fed with the input vector x.The term A(x) contains no adjustable parameters and satisfies the BC's.The second term G is constructed so as not to contribute to the BC's, since yt(x) satisfy them.This term can be formed by using a FFNN whose weights and biases are to be adjusted in order to deal with the minimization problem.

Computation of the Gradient
An efficient minimization of (2) can be considered as a procedure of training the FFNN, where the error corresponding to each input vector xi is the value E(xi) which has to forced near zero.Computation of this error value involves not only the FFNN output but also the derivatives of the output with respect to any of its inputs.Therefore, in computing the gradient of the error with respect to the network weights consider a multi layer FFNN with n input units (where n is the dimensions of the domain ) one hidden layer with H sigmoid units and a linear output unit .
For a given input vector x  ( x1, x2, …, xn ) the output of the FFNN is : wij denotes the weight connecting the input unit j to the hidden unit i vi denotes the weight connecting the hidden unit i to the out put unit , bi denotes the bias of hidden unit i, and σ (z) is the sigmoid transfer function ( tansig.).The gradient of FFNN, with respect to the parameters of the FFNN can be easily obtained as : N w Once the derivative of the error with respect to the network parameters has been defined, then it is a straight forward to employ any minimization technique.It must also be noted, the batch mode of weight updates may be employed.

Illustration Of The Method
In this section we describe solution of single BVP using FFNN .To illustrate the method, we will consider the 2nd order BVP : , (8) where N(x, p) is the output of a FFNN with one input unit for x and weights p .Note that yt(x) satisfies the BC by construction.The error quantity to be minimized is given by : + (x-a) (x-b) d2 N(x, p) /dx2 it is straightforward to compute the gradient of the error with respect to the parameters p using (4) - (6).The same holds for all subsequent model problems.Step9: Go to 7.

Example
In this section we report numerical result, we use a multi-layer FFNN having one hidden layer with 5 hidden units (neurons) and one linear output unit.The sigmoid activation of each hidden unit is tansig , the analytic solution ya(x) was known in advance.Therefore we test the accuracy of the obtained solutions by computing the deviation : y(x)  | yt(x) -ya(x) |.In order to illustrate the characteristics of the solutions provided by the neural network method, we provide figures displaying the corresponding deviation y(x) both at the few points (training points) that were used for training and at many other points (test points) of the domain of equation.The latter kind of figures are of major importance since they show the interpolation capabilities of the neural solution which to be superior compared to other solution obtained by using other methods.Moreover, we can consider points outside the training interval in order to obtain an estimate of the extrapolation performance of the obtained numerical solution.

Example 1
Consider the following 2nd order BVP : d2y/dx2 = -dy/dx + 2y with BC: y(0)  1 , y(1) = e and x  [0, 1].The analytic solution is : ya(x)  exp(x) , according to (8) the trial neural form of the solution is taken to be :  1) and its errors given in table (2), table (4) gives the weight and bias of the designer network ,table (3) gives the performance of the train with epoch and time .Ibraheem and Khalaf [7] solve this example by using (integration and interpolation techniques) and Neural Networks and gave the maximum error value is max  yexact -yNN  = 1.2089E-008 and solution time is 5.9070 sec.and the result obtained by the neural network given in figure 2 Example 2 Consider the following 2nd order BVP :  5) and its errors given in table (6), table (7) gives the weight and bias of the designer network ,table(8) gives the performance of the train with epoch and time .Ibraheem and Khalaf [7] solve this example by using (integration and interpolation techniques) and Neural Networks and gave the maximum error value is max  yexact -yNN  = 44.3729E-004and solution time is 3.8750 sec.and the result obtained by the neural network given in figure 4

Conclusion
From the above problems it is clear that the method which proposed can handle effectively ODE and provide accurate approximate solution throughout the whole domain and not only at the training points.As evident from the tables, the results of proposed method are more precise as compared to neural network suggested in [7].
Print to PDF without this message by purchasing novaPDF (http://www.novapdf.com/)It is very difficult to know which training algorithm will be the fastest for a given problem.It will depend on many factors including the complexity of the problem, the number of data points in the training set, the number of weights and biases in the FFNN, the error goal, and whether the FFNN is being used for pattern recognition (discriminant analysis) or function approximation (regression).In general, the practical results on FFNN show which contain up to a few hundred weights the Levenberg-Marquardt algorithm (trainlm) will have the fastest convergence, then trainbr and then trainbfg.However, "trainbr" it does not perform well on function approximation on problems.The "traincg", algorithms have relatively modest memory requirements in particular "traincgp", but the computation required does increase geometrically with the size of the FFNN .The performance of the various algorithms can be affected by the accuracy required of the approximation.

Table1: Analytic and Neural solution of example 1
Out where x  [a , b] and the BC : y(a)  A, y(b) = B, a trial solution can be written as : yt(x, p)  (bA-aB)/(b-a) + (B-A)x /(b-a) + (x-a)(x-b)N(x, p) where the xi  [a , b].Since : dyt(x, p)/dx  (B-A)/(b-a)+ {(x-a)+(xb)}N(x,p) + (x-a) (x-b) d N ( x , p ) d x  and Print to PDF without this message by purchasing novaPDF (http://www.novapdf.com/)d2yt(x, p) /dx2 = 2N(x, p) + 2{(x-a)+(x-b)} d N ( x , p ) d x  the main steps of the algorithm are the following: Step1: Determine the variable interval of the x, i.e. ( x  [a,b] ).Step2: input the analytic solution .Step3: Determine the Boundary condition.Step4: Determine the structure of the neural network for solving BVP.Step5: Determine the activation function and corresponding training algorithm Complete the design.Step6: Determine the trial solution.Step7: Implementation.Step8: compared the neural results and the exact results.Step9: stop after obtain the globally optimal solution.Step10: if no.

Table2 : Accuracy of solutions for example 1 Table3 : the performance of the train with epoch and time Table 4: Weight and bias of the network for different training algorithm
Print to PDF without this message by purchasing novaPDF (http://www.novapdf.com/)Deviationy(x) | y t (x)  y a (x) | w h e r e y t (x) computed by the following training algorithmPrint to PDF without this message by purchasing novaPDF (http://www.novapdf.com/)

Table 6 : Accuracy of solutions for example 2
Print to PDF without this message by purchasing novaPDF (http://www.novapdf.com/)