A new nonmonotone adaptive trust region line search method for unconstrained optimization

This paper proposes a new nonmonotone adaptive trust region line search method for solving unconstrained optimization problems, and presents a modified trust region ratio, which obtained more reasonable consistency between the accurate model and the approximate model. The approximation of Hessian matrix is updated by the modified BFGS formula. Trust region radius adopts a new adaptive strategy to overcome additional computational costs at each iteration. The global convergence and superlinear convergence of the method are preserved under suitable conditions. Finally, the numerical results show that the proposed method is very efficient.


Introduction
Consider the following unconstrained optimization problem where f : R n → R is a twice continuously differentiable function. Trust region method is one of prominent class of iterative methods. The basic idea of trust region methods as follows: at the current step x k , the trial step d k is obtained by solving the subproblem: where f k = f (x k ), g k = ∇f (x k ), G k = ∇ 2 f (x k ), B k be a symmetric approximation of G k , k is trust region radius, and · is the Euclidean norm.
To evaluate an agreement between the model and the objective function, the most ordinary ratio is defined as follows: where the numerator is called the actual reduction and the denominator is called the predicted reduction. The ratio ρ k is used to determine whether the trial step d k is accepted. Given μ ∈ [0, 1], if ρ k < μ, the trial step d k is not successful and the subproblem (2) should be resolved with a smaller radius. Otherwise, d k is acceptable and the radius should be increased.
It is well-known that monotone techniques may slow down the rate of convergence, especially in the presence of the narrow curved valley. The monotone techniques that require the objective function to be decreased at each iteration. In order to overcome these disadvantages, Grippo et al. [1] proposed a nonmonotone technique for Newton's method in 1986. In 1998, Nocedal and Yuan [2] proposed a nonmonotone trust region method with line search techniques, the step size α k satisfies the following inequality: where σ ∈ (0, 1). The general nonmonotone term However, the general nonmonotone strategy does not sufficiently employ the current value of the objective function f . It seems that the nonmonotone term has well performance far from the optimum. In order to introduce a more relaxed nonmonotone strategy, Ahookhosh et al. [3] introduced a modified nonmonotone term in 2002. More precisely, for σ ∈ (0, 1), the step size α k satisfies the following inequality: where the nonmonotone term R k is defined by in which η k ∈ [η min , η max ], with η min ∈ [0, 1), and η max ∈ [η min , 1]. One knows that an adaptive radius avoid the blindness of updating the initial trust region radius, and may cause the decrease in the total number of iterations. In 1997, Sartenear [4] proposed a new strategy for automatically determining the initial trust region radius. In 2002, Zhang et al. [5] proposed a new scheme to determine trust region radius as follows: k = c p B -1 k g k . To avoid calculating the inverse of the matrix B k and an estimation of B -1 k in each iteration, Li [6] proposed an adaptive trust region radius as follows: k = d k-1 y k-1 g k , where y k-1 = g kg k-1 . Inspired by these facts, some modified versions of adaptive trust region methods have been proposed in [7][8][9][10][11][12][13][14].
This paper is organized as follows. In Sect. 2, we describe the new algorithm. The global and superlinear convergence of the algorithm are established in Sect. 3. In Sect. 4, numerical results are reported, which show that the new method is effective. Finally, conclusions are drawn in Sect. 5.

New algorithm
In this section, a new adaptive nonmonotone trust region line search algorithm is proposed. Here, based on the method of Li [6], we proposed a adaptive trust region radius as follows: c k is an adjustment parameter. Prompted by the adaptive technique, the proposed method has the following well properties: it is convenient to adjust the radius by using the adjustment parameter c k , and the algorithm also reduces the related workload and calculation time.
On the basis of considered discussion, at each iteration, a trial step d k is obtained by solving the following trust region subproblem: where y k-1 = g kg k-1 . The matrix B k is updated by a modified BFGS formula [11], where Considering advantage of the Ahookhosh's nonmonotone term, the best convergence behavior can be obtained by adopting a stronger nonmonotone strategy away from the solution and a weaker monotone strategy closer to the solution. We defined a modified form of trust region ratio as follows: As seen, the effect of nonmonotonicity can be controlled in (10) by numerator and denominator. Now, we list the new adaptive nonmonotone trust region line search algorithm as follows:
Step 1. If g k ≤ ε, then stop. Otherwise, go to Step 2.
Step 2. Solve the subproblem (8) to obtain d k .
Step 3. Compute R k and ρ k respectively.
Step 5. If ρ k ≥ μ 1 , set x k+1 = x k + d k and go to Step 6. Otherwise, find the step size α k satisfying (5). Set x k+1 = x k + α k d k , go to Step 6. Step 6. Update the trust region radius by k+1 = c k+1 x k+1 -x k g k+1 -g k g k+1 and go to Step 7.
Step 7. Compute the new Hessian approximation B k+1 by a modified BFGS formula (9).
Set k = k + 1 and go to Step 1.
Remark 2.1 If f is a twice continuously differentiable function, then H1 implies that ∇f is continuous and uniformly bounded on Ω. Hence, there exists a constant L such that 3 Convergence analysis Lemma 3.1 There is a constant τ ∈ (0, 1), the trial step d k satisfies the following inequalities: Proof The proof is exactly similar to the proof of Lemma 6 and Lemma 7 of [15] and here is omitted.
where p k is the iteration of the solution to subproblem from the previous trial step d k-1 to the currently acceptable trial step d k .
Proof According to Step 4 of Algorithm 2.1, the trust region radius satisfies k ≥ c k is a feasible solution to trust region subproblem. Therefore, we obtain, Proof is a solution of subproblem (8) corresponding to p k = p. Firstly, we prove that ρ k ≥ μ 1 , for sufficiently large p. Using Lemma 3.1, Lemma 3.2 and Taylor's formula, we have Proof From (7) and d k ≤ k , we observe that Proof Set α = α k ρ , where ρ ∈ (0, 1). According to Step 5 of Algorithm 2.1 and (5), it is easy to show that Using the definition of R k and Taylor expansion, we have where ξ ∈ (x k , x k+1 ). Thus,we get On the other hand, form d k ≤ κ g k and (13), we can write Hence, combining above inequality and (20), we have Thus, we can obtain (18).
Proof From Lemma 3.3, we know that Algorithm 2.1 generates an infinite sequence {x k } satisfying ρ k ≥ μ 1 , we obtain, Then, Replacing k by l(k) -1, we can write Combine Lemma 3.8 with the above inequality, we get According to Assumption 2.1 and (12), we have where ω = τ κ min{1, 1 κM 1 }. It follows from (25) that The reminder of the proof is similar to a theorem of [1] and here is omitted.
On the basis of the above lemmas and analysis, we can obtain the global convergence result of Algorithm 2.1 as follows: Proof We assume that d k be the solution of subproblem (8) corresponding to p k = p, and we have an infinite sequence {x k } satisfying ρ k ≥ μ 1 .
According to Lemma 3.2, we have, This above inequality and Lemma 3.8 indicate that (27) holds.
We will prove the superlinear convergence of Algorithm 2.1 under suitable conditions.
Then the sequence {x k } converges to x * superlinearly, that is, Proof From (28) and d k ≤ k , we obtain Using Taylor expansion, there exists t k ∈ (0, 1) such that Thus, we can obtain that From (28) and ∇ 2 f (x * ) is Lipschitz continuous in a neighborhood of x * , we get Note that by Theorem 3.1, it is implied that and thus, we have d k → 0. We can obtain then, Combine ∇ 2 f (x * ) is a positive definite matrix and (33). Then, there exists a constant ς > 0, and k 0 ≥ 0 such that Thus, we obtain Combine above inequality with (31), we get lim k→∞

Preliminary numerical experiments
In this section, we perform numerical experiments on Algorithm 2.1. A set of unconstrained test problems are selected from [16]. The simulation experiment uses MAT-LAB 9.4, the processor uses Intel (R) Core (TM), 2.00 GHz, 6 GB RAM. Take exactly the same value for the public parameters of these algorithms: μ 1 = 0.25, μ 2 = 0.75, β 1 = 0.25, β 2 = 1.5, c 0 = 1, N = 5. The matrix B k is updated by (9). The stopping criterions are g k ≤ 10 -6 and the number of iterations exceeds 5000. We denote the number of gradient evaluations by "n i ", the number of function evaluations by "n f ".
For convenience, we use the following notations to represent the algorithms: SNTR: Standard nonmonotone trust region method [17]. ATRG: Nonmonotone Shi's adaptive trust region method with q k = -g k [18]. ATRN: Nonmonotone Shi's adaptive trust region method with q k = -B -1 k g k [18]. NLS: New nonmonotone adaptive trust region line search method. For standard nonmonotone trust region method, we update k by the following formula if ρ k ≥ μ 2 . Table 1 shows that the experiments were conducted to compare NLS and the standard trust region method with a different initial radius. One knows that an initial radius has a significant influence on the numerical results in the standard trust region methods. Moreover, the total number of iterations and function evaluations of the new algorithm are partly less than the standard nonmonotone trust region method. We also know that NLS outperforms with ATRG, ATRN respect to the total number of function evaluations and the total number of gradient evaluations. The performance profiles given by Dolan and More [19] are used to compare the efficiency of the three algorithms. Figures 1-2 give the performance profiles of the three algorithms for the number of function evaluations, and the number of gradient evaluations, respectively. As the figures show that Algorithm 2.1 grows up faster than the other algorithms. Therefore, we can deduce that the new algorithm is more efficient and robust than the other considered trust region algorithms for solving unconstrained optimization.

Conclusions
In this paper, a new nonmonotone adaptive trust region line search method is presented for unconstrained optimization problems. A new nonmonotone trust region ratio is introduced to enhance the effective of the algorithm. A new trust region radius is proposed, which relaxes the condition of accepting a trial step for the trust region methods. Theorem 3.1 and Theorem 3.2 have been shown that the proposed algorithm can preserve