Two New PRP Conjugate Gradient Algorithms for Minimization Optimization Models

Gonglin Yuan; Xiabin Duan; Wenjie Liu; Xiaoliang Wang; Zengru Cui; Zhou Sheng

doi:10.1371/journal.pone.0140071

Abstract

Two new PRP conjugate Algorithms are proposed in this paper based on two modified PRP conjugate gradient methods: the first algorithm is proposed for solving unconstrained optimization problems, and the second algorithm is proposed for solving nonlinear equations. The first method contains two aspects of information: function value and gradient value. The two methods both possess some good properties, as follows: 1)β_k ≥ 0 2) the search direction has the trust region property without the use of any line search method 3) the search direction has sufficient descent property without the use of any line search method. Under some suitable conditions, we establish the global convergence of the two algorithms. We conduct numerical experiments to evaluate our algorithms. The numerical results indicate that the first algorithm is effective and competitive for solving unconstrained optimization problems and that the second algorithm is effective for solving large-scale nonlinear equations.

Citation: Yuan G, Duan X, Liu W, Wang X, Cui Z, Sheng Z (2015) Two New PRP Conjugate Gradient Algorithms for Minimization Optimization Models. PLoS ONE 10(10): e0140071. https://doi.org/10.1371/journal.pone.0140071

Editor: Yongtang Shi, Nankai University, CHINA

Received: May 27, 2015; Accepted: September 20, 2015; Published: October 26, 2015

Copyright: © 2015 Yuan et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited

Data Availability: All data are available in the paper.

Funding: This work is supported by China NSF (Grant No. 11261006 and 11161003), NSFC No. 61232016, NSFC No. U1405254, the Guangxi Science Fund for Distinguished Young Scholars (No. 2015GXNSFGA139001) and PAPD issue of Jiangsu advantages discipline.

Competing interests: The authors have declared that no competing interests exist.

Introduction

As we know, the conjugate gradient method is very popular and effective for solving the following unconstrained optimization problem (1) where f : ℜⁿ → ℜ is continuously differentiable and g(x) denotes the gradient of f(x) at x, the problem Eq (1) also can be applied to model some other problems [1–5]. The iterative formula used in the conjugate gradient method is usually given by (2) and (3) where g_k = g(x_k), β_k ∈ ℜ is a scalar, α_k > 0 is a step length that is determined by some line search, and d_k denotes the search direction. Different conjugate methods have different choices for β_k. Some of the popular methods [6–12] used to compute β_k are the DY conjugate gradient method [6], FR conjugate gradient method [7], PRP conjugate gradient method [8, 9], HS conjugate gradient method [10], LS conjugate gradient method [11], and CD conjugate gradient method [12]. β_k [8, 9] is defined by (4) where denotes the Euclidean norm, y_k−1 = g_k−g_k−1. The PRP conjugate gradient method is currently considered to have the best numerical performance, but it does not have good convergence. With an exact line search, the global convergence of the PRP conjugate gradient method has been established by Polak and Ribière [8] for convex objective functions. However, Powell [13] proposed a counter example that proved the existence of nonconvex functions on which the PRP conjugate gradient method does not have global convergence, even with an exact line search. With the weak Wolfe-Powell line search, Gilbert and Nocedal [14] proposed a modified PRP conjugate gradient method by restricting β_k to be not less than zero and proved that it has global convergence, with the hypothesis that it satisfies the sufficient descent condition. Gilbert and Nocedal [14] also gave an example showing that β_k may be negative even though the objective function is uniformly convex. When the Strong Wolfe-Powell line search was used, Dai [15] gave a example showing that the PRP method cannot guarantee that every step search direction is the descent direction, even if the objective function is uniformly convex.

Through the above observations and [13, 14, 16–18], we know that the following sufficient descent condition (5) and the condition β_k is not less than zero are very important for establishing the global convergence of the conjugate gradient method.

The weak Wolfe-Powell (WWP) line search is designed to compute α_k and is usually used for the global convergence analysis. The WWP line search is as follows (6) and (7) where .

Recently, many new conjugate gradient methods ([19–28] etc.) that possess some good properties have been proposed for solving unconstrained optimization problems.

In Section 2, we state the motivation behind our approach and give a new modified PRP conjugate gradient method and new algorithm for solving problem Eq (1). In Section 3, we prove that the search direction of our new algorithm satisfies the sufficient descent property and trust region property; moreover, we establish the global convergence of the new algorithm with the WWP line search. In Section 4, we provide numerical experiment results for some test problems.

New algorithm for unconstrained optimization

Wei et al. [29] give a new PRP conjugate gradient method usually called the WYL method. When the WWP line search is used, this WYL method has global convergence under the sufficient descent condition. Zhang [30] give a modified WYL method called the NPRP method as follows

The NPRP method possesses better convergence properties. The above formula for y_k−1 contains only gradient value information, but some new y_k−1 formulas [31, 32] contain information on gradient value and function value. Yuan et al.[32] propose a new y_k−1 formula as follows and Where s_k−1 = x_k−x_k−1.

Li and Qu [33] give a modified PRP conjugate method as follows and

Under suitable conditions, Li and Qu [33] prove that the modified PRP conjugate method has global convergence.

Motivated by the above discussions, we propose a new modified PRP conjugate method as follows (8) and (9) where u₁ > 0, u₂ > 0, is the of [32].

As it follows directly from the above formula that . Next, we present a new algorithm and it’s diagram (Fig 1) as follows.

Download:

Fig 1. The diagram about Algorithm 2.1.

https://doi.org/10.1371/journal.pone.0140071.g001

Algorithm 2.1

Step 0: Given the initial point , set d₁ = −∇f(x₁) = −g₁, k: = 1.

Step 1: Calculate ; if , stop; otherwise, go to step 2.

Step 2: Calculate step length α_k by the WWP line search.

Step 3: Set x_k+1 = x_k + α_k d_k, then calculate ; if , stop; otherwise, go to step 4.

Step 4: Calculate the scalar β_k+1 by Eq (8) and calculate the search direction d_k+1 by Eq (9).

Step 5: Set k: = k + 1; go to step 2.

Global convergence analysis

Some suitable assumptions are often used to analyze the global convergence of the conjugate gradient method. Here, we state it as follows

Assumption 3.1

The level set Ω = {x ∈ ℜⁿ ∣ f(x) ≤ f(x₁)} is bounded.
In some neighborhood H of Ω, f is a continuously differentiable function, and the gradient function g of f is Lipschitz continuous, namely, there exists a constant L > 0 such that (10)

By Assumption 3.1, it is easy to obtain that there exist two constants A > 0 and η₁ > 0 satisfying (11)

Lemma 0.1 Let the sequence {d_k} be generated by Eq (9); then, we have (12)

Proof When k = 1, we can obtain by Eq (9), so Eq (12) holds. When k ≥ 2, we can obtain

The proof is achieved.

We know directly from above Lemma that our new method has the sufficient descent property.

Lemma 0.2 Let the sequence {x_k} and {d_k, g_k} be generated by Algorithm 2.1, and suppose that Assumption 3.1 holds; then, we can obtain (13)

Proof By Eq (7) and the Cauchy-Schwarz inequality, we have

Combining the above inequality with Assumption 3.1 ii) generates it is easy to know by lemma 0.1. By combining the above inequality with Eq (6), we obtain

Summing up the above inequalities from k = 1 to k = ∞, we can deduce that

By Eq (6), Assumption 3.1 and lemma 0.1, we know that {f_k} is bounded below, so we obtain

This finishes the proof.

The Eq (13) is usually called the Zoutendijk condition [34], and it is very important for establishing global convergence.

Lemma 0.3 Let the sequence {β_k, d_k} be generated by Algorithm 2.1, we have (14) where .

Proof When d_k = 0, we directly get g_k = 0 from Eq (12). When d_k ≠ 0, by the Cauchy-Schwarz inequality, we can easily obtain and

We can obtain

Using Eq (8), we have

Finally, when k ≥ 2 by Eq (9), we have

Let ; we obtain . This finishes the proof.

This lemma also shows that the search direction of our algorithm has the trust region property.

Theorem 0.1 Let the sequence {d_k, g_k, β_k} and {x_k} be generated by Algorithm 2.1. Suppose that Assumption 3.1 holds; then (15)

Proof By Eqs (12) and (13), we obtain (16)

By Eq (14), we have ; then, we obtain which together with Eq (16) can yield

From the above inequality, we can obtain . The proof is finished.

Numerical Results

When β_k+1 and d_k+1 are calculated by Eqs (4) and (3), respectively, in step 4 of Algorithm 2.1, we call it the PRP conjugate gradient algorithm. We test Algorithm 2.1 and the PRP conjugate gradient algorithm using some benchmark problems. The test environment is MATLAB 7.0, on a Windows 7 system. The initial parameters are given by We use the following Himmeblau stop rule, which satisfies

If ∣f(x_k)∣ ≤ ɛ₂, let stop1 = ; otherwise, let . The test program will be stopped if stop1 < ɛ₃ or is satisfied, where ɛ₂ = ɛ₃ = 10⁻⁶. When the total number of iterations is greater than one thousand, the test program will be stopped. The test results are given in Tables 1 and 2: x₁ denotes the initial point, Dim denotes the dimension of test function, NI denotes the the total number of iterations, and NFG = NF+NG (NF and NG denote the number of the function evaluations and the number of the gradient evaluations, respectively). denotes the function value when the program is stopped. The test problems are defined as follows.

Schwefel function:
Langerman function:
Schwefel′s function
Sphere function:
Griewangk function:
Rosenbrock function:
Ackley function:
Rastrigin function:

Download:

Table 1. Test results for Algorithm 2.1.

https://doi.org/10.1371/journal.pone.0140071.t001

Download:

Table 2. Test results for the PRP conjugate gradient algorithm.

https://doi.org/10.1371/journal.pone.0140071.t002

It is easy to see that the two algorithms are effective for the above eight test problems listed in Tables 1 and 2. We use the tool of Dolan and Morè [35] to analyze the numerical performance of the two algorithms.

For the above eight test problems, Fig 2 shows the numerical performance of the two algorithms when the information of NI is considered, and Fig 3 shows the the numerical performance of the two algorithms when the information of NFG is considered. From the above two figures, it is easy to see that Algorithm 2.1 yields a better numerical performance than the PRP conjugate gradient algorithm on the whole. From Tables 1 and 2 and the two figures, we can conclude that Algorithm 2.1 is effective and competitive for solving unconstrained optimization problems.

Download:

Fig 2. Performance profiles of the two algorithms (NI).

https://doi.org/10.1371/journal.pone.0140071.g002

Download:

Fig 3. Performance profiles of the two algorithms (NFG).

https://doi.org/10.1371/journal.pone.0140071.g003

A new algorithm is given for solving nonlinear equations in the next section. The sufficient descent property and the trust region property of the new algorithm are proved in Section 6; moreover, we establish the global convergence of the new algorithm. In Section 7, the numerical results are presented.

New algorithm for nonlinear equations

We consider the system of nonlinear equations (17) where q : ℜⁿ → ℜⁿ is a continuously differentiable and monotonic function. ∇q(x) denotes the Jacobian matrix of q(x); if ∇q(x) is symmetric, we call Eq (17) symmetric nonlinear equations. As q(x) is monotonic, the following inequality holds. If a norm function is defined as follows and we define the unconstrained optimization problem as follows, (18)

We know directly that the problem Eq (17) is equivalent to the problem Eq (18).

The iterative formula Eq (2) is also usually used in many algorithms for solving problem Eq (17). Many algorithms ([36–41], etc.) have been proposed for solving special classes of nonlinear equations. We are more interested in the process of dealing with large-scale nonlinear equations. By Eq (2), it is easy to see that the two factors of stepsize α_k and search direction d_k are very important for dealing with large-scale problems. When dealing with large-scale nonlinear equations and unconstrained optimization problems, there are many popular methods ([38, 42–46] etc.) for computing d_k, such as conjugate gradient methods, spectral gradient methods, and limited-memory quasi-Newton approaches. Some new line search methods [37, 47] have been proposed for calculating α_k. Li and Li [48] provide the following new derivative-free line search method (19) where α_k = max{γ, ργ, ρ² γ, …}, ρ ∈ (0,1), σ₃ > 0 and γ > 0. This line search method is very effective for solving large-scale nonlinear monotonic equations.

Solodov and Svaiter [49] presented a hybrid projection-proximal point algorithm that could conquer some drawbacks when the form Eq (18) is used with nonlinear equations. Yuan et al.[50] proposed a three-term PRP conjugate gradient algorithm by using the projection-based technique, which was introduced by Solodov et al.[51] for optimization problems. The projection-based technique is very effective for solving nonlinear equations. It involves certain methods to compute search direction d_k and certain line search methods to calculate α_k, which satisfies in which w_k = x_k + α_k d_k. For any x* that satisfies q(x*) = 0, considering that q(x) is monotonic, we can obtain

Thus, it is easy to obtain the current iterate x_k, which is strictly separated from the zeros of the system of equations Eq (17) by the following hyperplane

Then, the iterate x_k+1 can be obtained by projecting x_k onto the above hyperplane. The projection formula can be set as follows (20)

Yuan et al. [50] present a three-term Polak-Ribière-Polyak conjugate gradient algorithm in which the search direction d_k is defined as follows where y_k−1 = q_k−q_k−1. The derivative-free line search method [48] and the projection-based techniques are used by the algorithm [50], proved to be very suitable for solving large-scale nonlinear equations. The most attractive property of algorithm [50] is the the trust region property of d_k.

Motivated by our new modified PRP conjugate gradient formula, proposed in Section 2, we proposed the following modified PRP conjugate gradient formula (21) and (22) Where u₃ > 0, u₄ > 0. It is easy to see that , motivated by the above observation and [50]. We present a new algorithm for solving problem Eq (17): it uses our modified PRP conjugate gradient formula Eqs (21) and (22). Here, we list the new algorithm and it’s diagram (Fig 4) as follows.

Download:

Fig 4. The diagram about Algorithm 5.1.

https://doi.org/10.1371/journal.pone.0140071.g004

Algorithm 5.1

Step 1: Given the initial point x₁ ∈ ℜⁿ,ɛ₄ > 0,ρ ∈ (0,1), σ₃ > 0, γ > 0,u₃ > 0, u₄ > 0, and k: = 1.

Step 2: If stop; otherwise, go to step 3.

Step 3: Compute d_k by Eq (22) and calculate α_k by Eq (19)

Step 4: Set the next iterate to be w_k = x_k + α_k d_k;

Step 5: If , stop and set x_k+1 = w_k; otherwise, calculate x_k+1 by Eq (20)

Step 6: Set k: = k + 1; go to step 2.

Convergence Analysis

When we analyze the global convergence of Algorithm 5.1, we require the following suitable assumptions.

Assumption 6.1

The solution set of the problem Eq (17) is nonempty.
q(x) is Lipschitz continuous, namely, there exists a constant E > 0 such that

By Assumption 6.1, it is easy to obtain that there exists a positive constant ζ that satisfies (23)

Lemma 0.4 Let the sequence {d_k} be generated by Eq (22); then, we can obtain (24) and (25)

Proof As the proof is similar to Lemma 0.1 and Lemma 0.3 of this paper, we omit it here.

Similar to Lemma 3.1 of [50] and theorem 2.1 of [51], it is easy to obtain the following lemma. Here, we omit this proof and only list it.

Lemma 0.5 Suppose that Assumption 6.1 holds and x* is a solution of problem Eq (17) that satisfies g(x*) = 0. Let the sequence {x_k} be obtained by Algorithm 5.1; then, the {x_k} is a bounded sequence and holds. Moreover, either {x_k} is a infinite sequence and or the {x_k} is a finite sequence and a solution of problem Eq (17) is the last iteration.

Lemma 0.6 Suppose that Assumption 6.1 holds, then, an iteration x_k+1 = x_k + α_k d_k will be generated by Algorithm 5.1 in a finite number of backtracking steps.

Proof We will obtain this conclusion by contradiction: suppose that does not hold; then, there exists a positive constant ɛ₅ that satisfies (26) suppose that there exist some iterate indexes that do not satisfy the condition Eq (19). We let then it can obtain

By Assumption 6.1 (b) and Eq (24), we find

By Eqs (23) and (25), we can obtain

Thus, we obtain which shows that is bounded below. This contradicts the definition of ; so, the lemma holds.

Similar to Theorem 3.1 of [50], we list the following theorem but omit its proof.

Theorem 0.2 Let the sequence {x_k+1, q_k+1} and {α_k, d_k} be generated by Algorithm 5.1. Suppose that Assumption 6.1 holds; then, we have (27)

Numerical results

When the following d_k formula of the famous PRP conjugate gradient method [8, 9] is used to compute d_k in step 3 of Algorithm 5.1, then it is called PRP algorithm. We test Algorithm 5.1 and the PRP algorithm for some problems in this section. The test environment is MATLAB 7.0 on a Windows 7 system. The initial parameters are given by

When the number of iterations is greater than or equal to one thousand and five hundred, the test program will also be stopped. The test results are given in Tables 3 and 4. As we know, when the line search cannot guarantee that d_k satisfies , some uphill search direction may be produced; the line search method possibly fails in this case. In order to prevent this situation, when the search time is greater than or equal to fifteen in the inner cycle of our program, we set α_k that is acceptable. NG, NI stand for the number of gradient evaluations and iterations respectively. Dim denotes the dimension of the testing function, and cputime denotes the cpu time in seconds. GF denotes the evaluation of the final function norm when the program terminates. The test functions all have the following form the concrete function definitions are given as follows.

Function 1. Exponential function 2 Initial guess:
Function 2. Trigonometric function Initial guess:
Function 3. Logarithmic function Initial guess: x₀ = (1,1,⋯,1)^T.
Function 4. Broyden Tridiagonal function [[52], pp. 471–472] Initial guess: x₀ = (−1,−1,⋯,−1)^T.
Function 5. Strictly convex function 1 [[44], p. 29]
q(x) is the gradient of Initial guess:
Function 6. Variable dimensioned function Initial guess:
Function 7. Discrete boundary value problem [53]. Initial guess: x₀ = (h(h−1), h(2h−1),⋯,h(nh−1))^T.
Function 8. Troesch problem [54] Initial guess: x₀ = (0, 0, ⋯, 0)^T.

Download:

Table 3. Test results for Algorithm 5.1.

https://doi.org/10.1371/journal.pone.0140071.t003

Download:

Table 4. Test results for PRP algorithm.

https://doi.org/10.1371/journal.pone.0140071.t004

By Tables 3 and 4, we see that Algorithm 5.1 and the PRP algorithm are effective for solving the above eight problems.

We use the tool of Dolan and Morè [35] to analyze the numerical performance of the two algorithms when NI, NG and cputime are considered, for which we generate three figures.

Fig 5 shows that the numerical performance of Algorithm 5.1 is slightly better than that of the PRP algorithm when NI is considered. It is easy to see that the numerical performance of Algorithm 5.1 is better than that of the PRP algorithm from Figs 6 and 7 because the PRP algorithm requires a bigger horizontal axis when the problems are completely solved.

Download:

Fig 5. Performance profiles of the two algorithms (NI).

https://doi.org/10.1371/journal.pone.0140071.g005

Download:

Fig 6. Performance profiles of the two algorithms (NG).

https://doi.org/10.1371/journal.pone.0140071.g006

Download:

Fig 7. Performance profiles of the two algorithms (cputime).

https://doi.org/10.1371/journal.pone.0140071.g007

From the above two tables and three figures, we see that Algorithm 5.1 is effective and competitive for solving large-scale nonlinear equations.

Conclusion

(i) This paper provides the first new algorithm based on the first modified PRP conjugate gradient method in Sections 1–4. The β_k formula of the method includes the gradient value and function value. The global convergence of the algorithm is established under some suitable conditions. The trust region property and sufficient descent property of the method have been proved without the use of any line search method. For some test functions, the numerical results indicate that the first algorithm is effective and competitive for solving unconstrained optimization problems.

(ii) The second new algorithm based on the second modified PRP conjugate gradient method is presented in Sections 5-7. The new algorithm has global convergence under suitable conditions. The trust region property and the sufficient descent property of the method are proved without the use of any line search method. The numerical results of some tests function are demonstrated. The numerical results show that the second algorithm is very effective for solving large-scale nonlinear equations.

Acknowledgments

This work is supported by China NSF (Grant No. 11261006 and 11161003), NSFC No. 61232016, NSFC No. U1405254, the Guangxi Science Fund for Distinguished Young Scholars (No. 2015GXNSFGA139001) and PAPD issue of Jiangsu advantages discipline. The authors wish to thank the editor and the referees for their useful suggestions and comments which greatly improve this paper.

Author Contributions

Conceived and designed the experiments: GY XD WL. Performed the experiments: GY XD WL. Analyzed the data: GY XD WL XW ZC ZS. Contributed reagents/materials/analysis tools: GY XD WL XW ZC ZS. Wrote the paper: GY XD WL.

References

1. Gu B, Sheng V, Feasibility and finite convergence analysis for accurate on-line v-support vector learning, IEEE Transactions on Neural Networks and Learning Systems. 24 (2013) 1304–1315.
- View Article
- Google Scholar
2. Li J, Li X, Yang B, Sun X, Segmentation-based Image Copy-move Forgery Detection Scheme, IEEE Transactions on Information Forensics and Security. 10 (2015) 507–518.
- View Article
- Google Scholar
3. Wen X, Shao L, Fang W, Xue Y, Efficient Feature Selection and Classification for Vehicle Detection, IEEE Transactions on Circuits and Systems for Video Technology. 2015,
- View Article
- Google Scholar
4. Zhang H, Wu J, Nguyen TM, Sun M, Synthetic Aperture Radar Image Segmentation by Modified Student’s t-Mixture Model, IEEE Transaction on Geoscience and Remote Sensing. 52 (2014) 4391–4403.
- View Article
- Google Scholar
5. Fu Z, Achieving Efficient Cloud Search Services: Multi-keyword Ranked Search over Encrypted Cloud Data Supporting Parallel Computing, IEICE Transactions on Communications. E98-B (2015) 190–200.
- View Article
- Google Scholar
6. Dai Y, Yuan Y, A nonlinear conjugate gradient with a strong global convergence properties, SIAM J. Optim. 10 (2000) 177–182.
- View Article
- Google Scholar
7. Fletcher R, Reeves C, Function minimization by conjugate gradients, Comput. J. 7 (1964) 149–154.
- View Article
- Google Scholar
8. Polak E, Ribière G, Note sur la convergence de directions conjugees, Rev. Fran. Inf. Rech. Opérat. 3 (1969) 35–43.
- View Article
- Google Scholar
9. Polyak BT, The conjugate gradient method in extreme problems, USSR Comput. Math. Math.Phys. 9 (1969) 94–112.
- View Article
- Google Scholar
10. Hestenes MR, Stiefel EL, Methods of conjugate gradients for solving linear systems, J. Res. Natl.Bur. Stand. Sect. B. 49 (1952) 409–432.
- View Article
- Google Scholar
11. Liu Y, Storey C, Effcient generalized conjugate gradient algorithms part 1: theory. J. Comput. Appl.Math. 69 (1992) 17–41.
- View Article
- Google Scholar
12. Fletcher R, Practical method of optimization, vol I: unconstrained optimization, 2nd edn. Wiley,New York, 1997.
13. Powell MJD, Nonconvex minimization calculations and the conjugate gradient method, Lecture Notes in Mathematics, vol. 1066, Spinger, Berlin, 1984, pp. 122–141.
14. Gibert JC, Nocedal J, Global convergence properties of conjugate gradient methods for optimization. SIAM J. Optim. 2 (1992) 21–42.
- View Article
- Google Scholar
15. Dai Y, Analyses of conjugate gradient methods, PH.D.thesis, Institute of Computational Mathematics and Scientific/Engineering Computing, Chinese Academy of Sciences (in Chinese), 1997.
16. Ahmed T, Storey D, Efficient hybrid conjugate gradient techniques, Journal of Optimization Theory and Applications. 64 (1990) 379–394.
- View Article
- Google Scholar
17. Al-Baali A, Descent property and global convergence of the Flecher-Reeves method with inexact line search, IMA Journal of Numerical Analysis. 5 (1985) 121–124.
- View Article
- Google Scholar
18. Hu Y F, Storey C, Global convergence result for conjugate gradient methods, Journal of Optimization Theory and Applications. 71 (1991) 399–405.
- View Article
- Google Scholar
19. Yuan G, Wei Z, Zhao Q, A modified Polak-Ribière-Polyak conjugate gradient algorithm for large-scale optimization problems, IIE Transactions. 46 (2014), 397–413.
- View Article
- Google Scholar
20. Yuan G, Zhang M, A modified Hestenes-Stiefel conjugate gradient algorithm for large-scale optimization, Numerical Functional Analysis and Optimization. 34 (2013), 914–937.
- View Article
- Google Scholar
21. Yuan G, Lu X, Wei Z, A conjugate gradient method with descent direction for unconstrained optimization, Journal of Computational and Applied Mathematics, 233 (2009), 519–530.
- View Article
- Google Scholar
22. Yuan G, Lu X, A modified PRP conjugate gradient method, Annals of Operations Research. 166 (2009),73–90.
- View Article
- Google Scholar
23. Li X, Zhao X, A hybrid conjugate gradient method for optimization problems, Natural Science. 3(2011): 85–90.
- View Article
- Google Scholar
24. Yuan G, Modified nonlinear conjugate gradient methods with sufficient descent property for large-scale optimization problems, Optimization Letters. 3 (2009) 11–21.
- View Article
- Google Scholar
25. Huang H, Lin S, A modified Wei-Yao-Liu conjugate gradient method for unconstrained optimization, Applied Mathematics and Computation. 231 (2014) 179–186.
- View Article
- Google Scholar
26. Yu G, Zhao Y, Wei Z, A descent nonlinear conjugate gradient method for large-scale unconstrained optimization, Applied mathematics and computation. 187 (2) (2007) 636–643.
- View Article
- Google Scholar
27. Yao S, Wei Z, Huang H, A note about WYL’s conjugate gradient method and its applications, Applied Mathematics and computation. 191 (2) (2007) 381–388.
- View Article
- Google Scholar
28. Hager WW, Zhang H, A new conjugate gradient method with guaranteed descent and an efficient line search, SIAM Journal on Optimization. 16(1) (2005) 170–192.
- View Article
- Google Scholar
29. Wei Z, Yao S, Liu L, The convergence properties of some new conjugate gradient methods, Applied Mathematics and Computation. 183 (2006) 1341–1350.
- View Article
- Google Scholar
30. Zhang L, An improved Wei-Yao-Liu nonlinear conjugate gradient method for optimization computation, Applied Mathematics and computation. 215 (6) (2009) 2269–2274.
- View Article
- Google Scholar
31. Wei Z, Yu G, Yuan G, Lian Z, The superlinear convergence of a modified BFGS-type method for unconstrained optimization, Comput. Optim. Appl. 29 (2004) 315–332.
- View Article
- Google Scholar
32. Yuan G, Wei Z, Convergence analysis of a modified BFGS method on convex minimizations, Computational Optimization and Applications. 47 (2) (2010) 237–255.
- View Article
- Google Scholar
33. Li M, Qu A, Some sufficient descent conjugate gradient methods and their global convergence, Computational and Applied Mathematics. 33 (2) (2014) 333–347.
- View Article
- Google Scholar
34. Zoutendijk Z, Nonlinear programming computational methods. In: Abadie J. (ed.) Integer and nonlinear programming, North-Holland, Amsterdam, 1970, 37–86.
35. Dolan ED, Morè JJ, Benchmarking optimization software with performance profiles, Math.Program. 91 (2002) 201–213.
- View Article
- Google Scholar
36. Buhmiler S, Krejic´ N, Luzanin Z, Practical quasi-Newton algorithms for singular nonlinear systems, Numer. Algorithms 55 (2010) 481–502.
- View Article
- Google Scholar
37. Gu G, Li D, Qi L, Zhou S, Descent directions of quasi-Newton methods for symmetric nonlinear equations, SIAMJ.Numer.Anal. 40 (2002) 1763–1774.
- View Article
- Google Scholar
38. La Cruz W, Martínez. M, Raydan M. Spectral residual method without gradient information for solving large-scale nonlinear systems of equations, Math. Comp. 75 (2006) 1429–1448.
- View Article
- Google Scholar
39. Kanzow C, Yamashita N, Fukushima M, Levenberg-Marquardt methods for constrained nonlinear equations with strong local convergence properties, J. Comput. Appl. Math. 172 (2004) 375–397.
- View Article
- Google Scholar
40. Zhang J, Wang Y, A new trust region method for nonlinear equations, Math. Methods Oper. Res,58 (2003) 283–298.
- View Article
- Google Scholar
41. Grippo L, Sciandrone M, Nonmonotone derivative-free methods for nonlinear equations, Comput. Optim. Appl. 37 (2007) 297–328.
- View Article
- Google Scholar
42. Cheng W, A PRP type method for systems of monotone equations, Math. Comput. Modelling. 50 (2009) 15–20.
- View Article
- Google Scholar
43. La Cruz W, Raydan M, Nonmonotone spectral methods for large-scale nonlinear systems, Optim. Methods Softw. 18 (2003) 583–599.
- View Article
- Google Scholar
44. Raydan M, The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem, SIAM J. Optim. 7 (1997) 26–33.
- View Article
- Google Scholar
45. Yu G, Guan L, Chen W, Spectral conjugate gradient methods with sufficient descent property for large-scale unconstraned optimization, Optim.Methods Softw. 23 (2) (2008) 275–293.
- View Article
- Google Scholar
46. Yuan G, Wei Z, Lu S, Limited memory BFGS method with backtracking for symmetric nonlinear equations, Math. Comput. Modelling. 54 (2011) 367–377.
- View Article
- Google Scholar
47. Li D, Fukushima M, A global and superlinear convergent Gauss-Newton -based BFGS method for symmetric nonlinear equations, SIAMJ.Numer.Anal. 37 (1999) 152–172.
- View Article
- Google Scholar
48. Li Q, Li D, A class of derivative-free methods for large-scale nonlinear monotone equations, IMA J. Numer. Anal. 31 (2011) 1625–1635.
- View Article
- Google Scholar
49. Solodov MV, Svaiter BF, A hybrid projection-proximal point algorithm, J. Convex Anal.6 (1999) 59–70.
- View Article
- Google Scholar
50. Yuan G, Zhang M, A three-terms Polak-Ribière-Polyak conjugate gradient algorithm for large-scale nonlinear equations, Journal of Computational and Applied Mathematics. 286 (2015) 186–195.
- View Article
- Google Scholar
51. Solodov MV, Svaiter BF, A globally convergent inexact Newton method for systems of monotone equations, in: Fukushima M., Qi L. (Eds.),Reformulation:Nonsmooth, Piecewise Smooth, Semismooth and Smoothing Methods, Kluwer Academic Publishers. 1998, pp. 355–369.
52. Gomez-Ruggiero M, Martinez J, Moretti A, Comparing algorithms for solving sparse nonlinear systems of equations, SIAM J. Sci. Comput. 23 (1992)459–483.
- View Article
- Google Scholar
53. Morè J, Garbow B, Hillström K, Testing unconstrained optimization software, ACM Trans. Math. Softw. 7 (1981) 17–41.
- View Article
- Google Scholar
54. Roberts SM, Shipman JJ, On the closed form solution of Troesch’s problem, J. Comput. Phys. 21 (1976) 291–304.
- View Article
- Google Scholar

[ref1] 1. Gu B, Sheng V, Feasibility and finite convergence analysis for accurate on-line v-support vector learning, IEEE Transactions on Neural Networks and Learning Systems. 24 (2013) 1304–1315.
View Article
Google Scholar

[2] View Article

[3] Google Scholar

[ref2] 2. Li J, Li X, Yang B, Sun X, Segmentation-based Image Copy-move Forgery Detection Scheme, IEEE Transactions on Information Forensics and Security. 10 (2015) 507–518.
View Article
Google Scholar

[5] View Article

[6] Google Scholar

[ref3] 3. Wen X, Shao L, Fang W, Xue Y, Efficient Feature Selection and Classification for Vehicle Detection, IEEE Transactions on Circuits and Systems for Video Technology. 2015,
View Article
Google Scholar

[8] View Article

[9] Google Scholar

[ref4] 4. Zhang H, Wu J, Nguyen TM, Sun M, Synthetic Aperture Radar Image Segmentation by Modified Student’s t-Mixture Model, IEEE Transaction on Geoscience and Remote Sensing. 52 (2014) 4391–4403.
View Article
Google Scholar

[11] View Article

[12] Google Scholar

[ref5] 5. Fu Z, Achieving Efficient Cloud Search Services: Multi-keyword Ranked Search over Encrypted Cloud Data Supporting Parallel Computing, IEICE Transactions on Communications. E98-B (2015) 190–200.
View Article
Google Scholar

[14] View Article

[15] Google Scholar

[ref6] 6. Dai Y, Yuan Y, A nonlinear conjugate gradient with a strong global convergence properties, SIAM J. Optim. 10 (2000) 177–182.
View Article
Google Scholar

[17] View Article

[18] Google Scholar

[ref7] 7. Fletcher R, Reeves C, Function minimization by conjugate gradients, Comput. J. 7 (1964) 149–154.
View Article
Google Scholar

[20] View Article

[21] Google Scholar

[ref8] 8. Polak E, Ribière G, Note sur la convergence de directions conjugees, Rev. Fran. Inf. Rech. Opérat. 3 (1969) 35–43.
View Article
Google Scholar

[23] View Article

[24] Google Scholar

[ref9] 9. Polyak BT, The conjugate gradient method in extreme problems, USSR Comput. Math. Math.Phys. 9 (1969) 94–112.
View Article
Google Scholar

[26] View Article

[27] Google Scholar

[ref10] 10. Hestenes MR, Stiefel EL, Methods of conjugate gradients for solving linear systems, J. Res. Natl.Bur. Stand. Sect. B. 49 (1952) 409–432.
View Article
Google Scholar

[29] View Article

[30] Google Scholar

[ref11] 11. Liu Y, Storey C, Effcient generalized conjugate gradient algorithms part 1: theory. J. Comput. Appl.Math. 69 (1992) 17–41.
View Article
Google Scholar

[32] View Article

[33] Google Scholar

[ref12] 12. Fletcher R, Practical method of optimization, vol I: unconstrained optimization, 2nd edn. Wiley,New York, 1997.

[ref13] 13. Powell MJD, Nonconvex minimization calculations and the conjugate gradient method, Lecture Notes in Mathematics, vol. 1066, Spinger, Berlin, 1984, pp. 122–141.

[ref14] 14. Gibert JC, Nocedal J, Global convergence properties of conjugate gradient methods for optimization. SIAM J. Optim. 2 (1992) 21–42.
View Article
Google Scholar

[37] View Article

[38] Google Scholar

[ref15] 15. Dai Y, Analyses of conjugate gradient methods, PH.D.thesis, Institute of Computational Mathematics and Scientific/Engineering Computing, Chinese Academy of Sciences (in Chinese), 1997.

[ref16] 16. Ahmed T, Storey D, Efficient hybrid conjugate gradient techniques, Journal of Optimization Theory and Applications. 64 (1990) 379–394.
View Article
Google Scholar

[41] View Article

[42] Google Scholar

[ref17] 17. Al-Baali A, Descent property and global convergence of the Flecher-Reeves method with inexact line search, IMA Journal of Numerical Analysis. 5 (1985) 121–124.
View Article
Google Scholar

[44] View Article

[45] Google Scholar

[ref18] 18. Hu Y F, Storey C, Global convergence result for conjugate gradient methods, Journal of Optimization Theory and Applications. 71 (1991) 399–405.
View Article
Google Scholar

[47] View Article

[48] Google Scholar

[ref19] 19. Yuan G, Wei Z, Zhao Q, A modified Polak-Ribière-Polyak conjugate gradient algorithm for large-scale optimization problems, IIE Transactions. 46 (2014), 397–413.
View Article
Google Scholar

[50] View Article

[51] Google Scholar

[ref20] 20. Yuan G, Zhang M, A modified Hestenes-Stiefel conjugate gradient algorithm for large-scale optimization, Numerical Functional Analysis and Optimization. 34 (2013), 914–937.
View Article
Google Scholar

[53] View Article

[54] Google Scholar

[ref21] 21. Yuan G, Lu X, Wei Z, A conjugate gradient method with descent direction for unconstrained optimization, Journal of Computational and Applied Mathematics, 233 (2009), 519–530.
View Article
Google Scholar

[56] View Article

[57] Google Scholar

[ref22] 22. Yuan G, Lu X, A modified PRP conjugate gradient method, Annals of Operations Research. 166 (2009),73–90.
View Article
Google Scholar

[59] View Article

[60] Google Scholar

[ref23] 23. Li X, Zhao X, A hybrid conjugate gradient method for optimization problems, Natural Science. 3(2011): 85–90.
View Article
Google Scholar

[62] View Article

[63] Google Scholar

[ref24] 24. Yuan G, Modified nonlinear conjugate gradient methods with sufficient descent property for large-scale optimization problems, Optimization Letters. 3 (2009) 11–21.
View Article
Google Scholar

[65] View Article

[66] Google Scholar

[ref25] 25. Huang H, Lin S, A modified Wei-Yao-Liu conjugate gradient method for unconstrained optimization, Applied Mathematics and Computation. 231 (2014) 179–186.
View Article
Google Scholar

[68] View Article

[69] Google Scholar

[ref26] 26. Yu G, Zhao Y, Wei Z, A descent nonlinear conjugate gradient method for large-scale unconstrained optimization, Applied mathematics and computation. 187 (2) (2007) 636–643.
View Article
Google Scholar

[71] View Article

[72] Google Scholar

[ref27] 27. Yao S, Wei Z, Huang H, A note about WYL’s conjugate gradient method and its applications, Applied Mathematics and computation. 191 (2) (2007) 381–388.
View Article
Google Scholar

[74] View Article

[75] Google Scholar

[ref28] 28. Hager WW, Zhang H, A new conjugate gradient method with guaranteed descent and an efficient line search, SIAM Journal on Optimization. 16(1) (2005) 170–192.
View Article
Google Scholar

[77] View Article

[78] Google Scholar

[ref29] 29. Wei Z, Yao S, Liu L, The convergence properties of some new conjugate gradient methods, Applied Mathematics and Computation. 183 (2006) 1341–1350.
View Article
Google Scholar

[80] View Article

[81] Google Scholar

[ref30] 30. Zhang L, An improved Wei-Yao-Liu nonlinear conjugate gradient method for optimization computation, Applied Mathematics and computation. 215 (6) (2009) 2269–2274.
View Article
Google Scholar

[83] View Article

[84] Google Scholar

[ref31] 31. Wei Z, Yu G, Yuan G, Lian Z, The superlinear convergence of a modified BFGS-type method for unconstrained optimization, Comput. Optim. Appl. 29 (2004) 315–332.
View Article
Google Scholar

[86] View Article

[87] Google Scholar

[ref32] 32. Yuan G, Wei Z, Convergence analysis of a modified BFGS method on convex minimizations, Computational Optimization and Applications. 47 (2) (2010) 237–255.
View Article
Google Scholar

[89] View Article

[90] Google Scholar

[ref33] 33. Li M, Qu A, Some sufficient descent conjugate gradient methods and their global convergence, Computational and Applied Mathematics. 33 (2) (2014) 333–347.
View Article
Google Scholar

[92] View Article

[93] Google Scholar

[ref34] 34. Zoutendijk Z, Nonlinear programming computational methods. In: Abadie J. (ed.) Integer and nonlinear programming, North-Holland, Amsterdam, 1970, 37–86.

[ref35] 35. Dolan ED, Morè JJ, Benchmarking optimization software with performance profiles, Math.Program. 91 (2002) 201–213.
View Article
Google Scholar

[96] View Article

[97] Google Scholar

[ref36] 36. Buhmiler S, Krejic´ N, Luzanin Z, Practical quasi-Newton algorithms for singular nonlinear systems, Numer. Algorithms 55 (2010) 481–502.
View Article
Google Scholar

[99] View Article

[100] Google Scholar

[ref37] 37. Gu G, Li D, Qi L, Zhou S, Descent directions of quasi-Newton methods for symmetric nonlinear equations, SIAMJ.Numer.Anal. 40 (2002) 1763–1774.
View Article
Google Scholar

[102] View Article

[103] Google Scholar

[ref38] 38. La Cruz W, Martínez. M, Raydan M. Spectral residual method without gradient information for solving large-scale nonlinear systems of equations, Math. Comp. 75 (2006) 1429–1448.
View Article
Google Scholar

[105] View Article

[106] Google Scholar

[ref39] 39. Kanzow C, Yamashita N, Fukushima M, Levenberg-Marquardt methods for constrained nonlinear equations with strong local convergence properties, J. Comput. Appl. Math. 172 (2004) 375–397.
View Article
Google Scholar

[108] View Article

[109] Google Scholar

[ref40] 40. Zhang J, Wang Y, A new trust region method for nonlinear equations, Math. Methods Oper. Res,58 (2003) 283–298.
View Article
Google Scholar

[111] View Article

[112] Google Scholar

[ref41] 41. Grippo L, Sciandrone M, Nonmonotone derivative-free methods for nonlinear equations, Comput. Optim. Appl. 37 (2007) 297–328.
View Article
Google Scholar

[114] View Article

[115] Google Scholar

[ref42] 42. Cheng W, A PRP type method for systems of monotone equations, Math. Comput. Modelling. 50 (2009) 15–20.
View Article
Google Scholar

[117] View Article

[118] Google Scholar

[ref43] 43. La Cruz W, Raydan M, Nonmonotone spectral methods for large-scale nonlinear systems, Optim. Methods Softw. 18 (2003) 583–599.
View Article
Google Scholar

[120] View Article

[121] Google Scholar

[ref44] 44. Raydan M, The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem, SIAM J. Optim. 7 (1997) 26–33.
View Article
Google Scholar

[123] View Article

[124] Google Scholar

[ref45] 45. Yu G, Guan L, Chen W, Spectral conjugate gradient methods with sufficient descent property for large-scale unconstraned optimization, Optim.Methods Softw. 23 (2) (2008) 275–293.
View Article
Google Scholar

[126] View Article

[127] Google Scholar

[ref46] 46. Yuan G, Wei Z, Lu S, Limited memory BFGS method with backtracking for symmetric nonlinear equations, Math. Comput. Modelling. 54 (2011) 367–377.
View Article
Google Scholar

[129] View Article

[130] Google Scholar

[ref47] 47. Li D, Fukushima M, A global and superlinear convergent Gauss-Newton -based BFGS method for symmetric nonlinear equations, SIAMJ.Numer.Anal. 37 (1999) 152–172.
View Article
Google Scholar

[132] View Article

[133] Google Scholar

[ref48] 48. Li Q, Li D, A class of derivative-free methods for large-scale nonlinear monotone equations, IMA J. Numer. Anal. 31 (2011) 1625–1635.
View Article
Google Scholar

[135] View Article

[136] Google Scholar

[ref49] 49. Solodov MV, Svaiter BF, A hybrid projection-proximal point algorithm, J. Convex Anal.6 (1999) 59–70.
View Article
Google Scholar

[138] View Article

[139] Google Scholar

[ref50] 50. Yuan G, Zhang M, A three-terms Polak-Ribière-Polyak conjugate gradient algorithm for large-scale nonlinear equations, Journal of Computational and Applied Mathematics. 286 (2015) 186–195.
View Article
Google Scholar

[141] View Article

[142] Google Scholar

[ref51] 51. Solodov MV, Svaiter BF, A globally convergent inexact Newton method for systems of monotone equations, in: Fukushima M., Qi L. (Eds.),Reformulation:Nonsmooth, Piecewise Smooth, Semismooth and Smoothing Methods, Kluwer Academic Publishers. 1998, pp. 355–369.

[ref52] 52. Gomez-Ruggiero M, Martinez J, Moretti A, Comparing algorithms for solving sparse nonlinear systems of equations, SIAM J. Sci. Comput. 23 (1992)459–483.
View Article
Google Scholar

[145] View Article

[146] Google Scholar

[ref53] 53. Morè J, Garbow B, Hillström K, Testing unconstrained optimization software, ACM Trans. Math. Softw. 7 (1981) 17–41.
View Article
Google Scholar

[148] View Article

[149] Google Scholar

[ref54] 54. Roberts SM, Shipman JJ, On the closed form solution of Troesch’s problem, J. Comput. Phys. 21 (1976) 291–304.
View Article
Google Scholar

[151] View Article

[152] Google Scholar

Figures

Abstract

Introduction

New algorithm for unconstrained optimization

Global convergence analysis

Numerical Results

New algorithm for nonlinear equations

Convergence Analysis

Numerical results

Conclusion

Acknowledgments

Author Contributions

References