Positive Definiteness of Symmetric Rank 1 (H-Version) Update for Unconstrained Optimization

: Several attempts have been made to modify the quasi-Newton condition in order to obtain rapid convergence with complete properties (symmetric and positive definite) of the inverse of Hessian matrix (second derivative of the objective function). There are many unconstrained optimization methods that do not generate positive definiteness of the inverse of Hessian matrix. One of those methods is the symmetric rank 1( H-version) update (SR1 update), where this update satisfies the quasi-Newton condition and the symmetric property of inverse of Hessian matrix, but does not preserve the positive definite property of the inverse of Hessian matrix where the initial inverse of Hessian matrix is positive definiteness. The positive definite property for the inverse of Hessian matrix is very important to guarantee the existence of the minimum point of the objective function and determine the minimum value of the objective


Introduction:
Symmetric Rank 1 (SR1 H-version) update is important in theoretical research and practical computing. However, the drawback that SR1 (Hversion) update does not retain the positive definiteness of updates hurts its performance in computing 1 . Fortunately, the drawback can be avoided if the modified quasi-Newton condition has been employed to modify the SR1 (H-version) update.
Zhang and Ch 2 introduced the modified quasi-Newton equation which uses both gradient and function value information in order to yield a higher order accuracy for approximating the second curvature of an objective function. Yabe, H and M 3 considered a modified Broyden family which includes the BFGS (Broyden-Fletcher-Goldfarb-Shanno) update. Guo and J 4 modified the BFGS update based on the new quasi-Newton equation, where is a matrix. Mahmood and H 5 Introduced the modified DFP (Davidon-Fletcher-Powell) update based on Zhang-Xu's condition and provided the global and superlinear convergence of the proposed method. Mahmood and S 6 proposed a modified Broyden update based on the positive definite property of Hessian matrix, via updating the vector y ( the difference between the next gradient of the objective function and the current gradient of the objective function) and provided the global and superlinear convergence of the proposed method. Razieh,B and H 7 introduced the modified BFGS method for solving the system of non-linear equations by using Taylor theorem, this proposed method is derivative-free, so the gradient information is not needed at each iteration. Razieh, B and H 8 proposed a modified quasi-Newton equation to get a more accurate approximation of the second curvature of the objective function by using Chain rule. Then, based on this modified secant equation, they present a new BFGS method for solving unconstrained optimization problems. Bojari and R 9 proposed a new family of modified BFGS update to solve the unconstrained optimization problem for nonconvex functions based on a new modified weak Wolfe -Powell line search technique. Yuan 10 proposed a modified BFGS algorithm which requires that the function value is matched, instead of the gradient value, at the previous iterate. This new algorithm preserves the global and local superlinear convergence properties of the BFGS algorithm.
In this research a modified update for the SR1 (H-version) update has been proposed to guarantees the positive definite property and preserves the symmetry property for the inverse of Hessian matrix via updating the vector s which represents the difference between the next solution and the current solution. The proof of convergence for the proposed method is given, and then tow numerical examples has been solved by the original SR1 (H-version) update and also solved by the proposed method.

Modified SRI (H-version) Update:
In this section the positive definite property for the inverse of Hessian matrix has been guarantee by updating the vector . Then for this purpose let us consider the objective function : → with the following assumptions: i. is twice continuously differentiable. ii. is uniformly convex, i.e. ∃ 1 , 2 ∈ + ∋ 1 ‖ ‖ 2 ≤ ∇ 2 ≤ 2 ‖ ‖ 2 ,∀ ∈ SR1 update (7) try to update the Hessian matrix by using the formula , and by using Sherman-Morrison-Woodbury formula 11 , the inverse of the Hessian matrix can be write as Which represent the solution of the quasi-Newton condition 12 +1 = (2) Where +1 is the next Hessian matrix, is the current Hessian matrix, +1 is the next inverse of Hessian matrix, is the current Hessian matrix, is the difference between the current solution and the next solution( = +1 − ), and is the difference between the current gradient and the next gradient of the objective function ( = ∇ ( +1 ) − ∇ ( )).
Eq. 1 does not preserve the positive definite property because if ( − ) < 0 then, +1 is not always positive for all ∈ , that means there is no guarantee to minimize the objective function at each iteration, so if the current inverse of Hessian matrix is positive definite then, the next inverse of Hessian matrix may be not positive definite and hence this iteration must be deleted. Now define: where ∈ , and form Eq. 2 This is called the modified quasi-Newton condition. The formula of inverse of Hessian matrix for the SR1 (H-version) update has been considered with replacing each by , in Eq. 1, and hence: and by substitution Eq. 6 in Eq. 5, then Now, set > 0 and by Eq. In addition, by more simplifying from Eq. 5 and Eq. 8, +1 can be write as follows: This is called the modified SR1(H-version) update. The sequence of inverse Hessian matrix produced by Eq. 9, never go to a near singular matrix which make the computation never break before get the minimizer of the objective function. Let 0 ≠ ∈ , then By substitution Eq. 6 in Eq. 10, Since > 0, and > 0 by the positive definiteness of , and ‖ ‖ is always positive, therefore, +1 > 0 and +1 is positive definite.

Numerical Examples:
In this section, two numerical examples are studied by modifying SR1 (H-version) update. The results are compared with the results obtained by the original method.

Example 1:
In this example the objective function ( ) = (1 − 1 ) 2 + ( 2 − 1 ) 2 1 , has been solved by using the original method firstly, and then also solved by our method. Min.  f =0.25), because of the non-positive definite of inverse of Hessian matrix 1 generated in first iteration (| 1 | = 0), and hence the method terminated at a saddle point which is not minimizer of the objective function, but clear that the modified SR1 (H-version) update can terminate successfully at the minimizer of the objective function. Moreover, the inverse of Hessian matrix generated by our method is positively definite at every iteration. From example 2, the function evaluation (FVAL) at the last iteration for the original method is very far from the exact value (min. f = 0), but in our method, it is clear that the function evaluation (FVAL) at the last iteration is very closely to the exact value ( min. f =0). This means that the original method cannot successfully terminate at the minimum because of the not positive definite of inverse Hessian matrix generated by the method in iteration number 43 where INVHESSIAN= 1.0e-005 * -0.1227 0.2728 0.2728 -0.3763 is very closely to zero (| 43 | = 0.0000000176 ) or near singular matrix.

Conclusion:
In this paper, the SR1 (H-version) update has been modified to preserve the positive definite property for the next inverse of Hessian matrix at each iteration if the current inverse of Hessian matrix is positively definite which makes the computation continue until the objective function terminates at the minimum of the objective function. Moreover, theorem 1 proves the positive definiteness property and theorem 5 proves the convergence of our method and also two numerical examples are established to support our method.