1. Introduction
We consider the unconstrained optimization problem
where f . Rn → R is a continuously differentiable function. The nonlinear conjugate gradient method is very useful for solving (1.1), especially when n is large. For any given x ∈ Rn, the nonlinear conjugate gradient method generates xk, k = 1, 2,...,n, by the following recursive relation
where gk = ∇f(xk) is the gradient of f at xk and βk is typically given by some formulas (such as [1-5]).
To achieve good computational performance and maintain the attractive feature of strong global convergence, in the past years, there exist many hybridizations of the basic conjugate gradient methods (see [6-10]). Based on the above papers, in this paper, we present a new hybridization nonlinear conjugate gradient method, where βk is given as
αk is computed by the Wolfe-type line search, which is proposed in [11]
where 0 <, δ <, σ <, 1, 0 <, γ <, 1. Based on the hybridization of βk as given by (1.4) we give the nonlinear conjugate gradient methods under the Wolfe-type line search and the general Wolfe line search.
In Section 2, we give the Method 2.1 and prove the global convergence of the proposed method with Wolfe-type line search. In Section 3, some discussions and the numerical results of the Method 2.1 are also given.
2. Method 2.1 and its global convergence analysis
Now, we give the Method 2.1 for solving (1.1).
Method 2.1
Step 1. Choose initial point x0 ∈ Rn, ε ≥ 0, 0 <, δ <, σ <, 1, u, γ ∈ (0, 1).
Step 2. Set d1 = −g1, k = 1, if ║g1║ = 0, then stop.
Step 3. Let xk+1 = xk + αkdk, compute αk by (1.5) and (1.6).
Step 4. Compute gk+1, if ║gk+1║ ≤ ε, then stop. Otherwise, go to next step.
Step 5. Compute βk+1 by (1.4) and generate dk+1 by (1.3).
Step 6. Set k = k + 1, go to step 3.
In order to establish the global convergence of the Method 2.1, we need the following assumption, which are often used in the literature to analyze the global convergence of nonlinear conjugate gradient methods.
Assumption 2.2
(i) The level set Ω = {x ∈ Rn|f(x) ≤ f(x1)} is bounded, i.e., there exists a positive constant C such that ║x║ ≤ C, for all x ∈ Ω.
(ii) In some neighborhood Ω of L, f is continuously differentiable and its gradient is Lipchitz continuous, i.e., there exists a constant L > 0, such that
for all x,y ∈ Ω.
Theorem 2.1. Let the sequences {xk} and {dk} be generated by the method (1.2), (1.3), and βk is computed by (1.4). Then, we have
for all k ≥ 1, where u ∈ (0, 1).
Proof. If k = 1, from (1.3), we get
Then, we can easily conclude (2.1). If k ≥ 2, multiplying (1.3) by , from (1.4), we get
Theorem 2.2. Suppose that Assumption 2.2 holds. By the Method 2.1, we have
Proof. From Theorem 2.1 and Assumption (i), we can know that {f(xk)} is bounded and monotonically decreasing, i.e., {f(xk)}, k = 1, 2,...,n, is convergent series. By (1.6), we have that
From Assumption 2.2, we get
So, from (2.3) and (2.4), we have
Square both sides of (2.5), we have
Therefore, by , we get
According to the convergence of {f(xk)}, we can conclude that
Remark 2.3. Suppose that Assumption 2.2 holds. By the Method 2.1, we know that
Proof. From Theorem 2.1, we know that
where u ∈ (0, 1).
Square both sides of (2.7), we have
Divided both sides of the above inequation by ║dk║2 , we get
From Theorem 2.2, we can conclude that
Theorem 2.4. Suppose that Assumption 2.2 holds. If {xk} (k = 1, 2,...,n) is generated by Method 2.1, we have
Proof. If (2.8) does not hold, there exists , such that
holds for all k ≥ 1. From (1.4), if , we have
From (1.3) and (1.4), we know that
Square both sides of (2.10) , we get
Divided both sides of the above equation by , we get
By
we have
By (2.9) and (2.11), we know that
Therefore, by , we have
If , we get
We can easily conclude that
which leads to a contradiction with (2.2). This shows (2.8) holds. We finish the proof of the theorem. □
3. Discussions of the Method 2.1 and Numerical Results
The line search in Method 2.1 can also given by the general Wolfe line search
where 0 <, σ1 ≤ σ2 <, 1.
Theorem 3.1. Suppose that Assumption 2.2 holds. Consider the Method 2.1, where αk satisfies (3.1), (3.2). Then, we have
Proof. From (3.2), we get
So
From (3.1), we have
By (3.3), we get
That is
We have
Discussion 3.1 By Theorem 3.1, we also can get the global convergence of the Method 2.1 with (3.1), (3.2).
Discussion 3.2 If the line search in the Method 2.1 is given by the other Wolfe-type line search, which is given in [12]
the method is also globally convergent.
Discussion 3.3 If in the Method 2.1, βk is given as
αk satisfies (1.5), (1.6) or (3.1), (3.2), we also can get the global convergence of the Method 2.1.
Numerical Results 3.4
Now, we test the Method 2.1, where αk satisfing (1.5), (1.6) or (3.1), (3.2) by using double precision versions of the unconstrained optimization problems in the CUTE library [13].
For the Method 2.1, αk is computed by (1.5) and (1.6) with δ = 0.4 and σ = 0.7 in the Table 3.1. αk is computed by (3.1) and (3.2) with σ1 = 0.5 and σ2 = 0.6 in the Table 3.2.
Table 3.1
Table 3.2
The numerical results are given in the form of NI/NF/NG/g, where NI, NF, NG denote the numbers of iterations, function evaluations, and gradient evaluations and g denotes the finally gradient norm. Finally, all attempts to solve the test problems were limited to reaching maximum of achieving a solution with ║gk║ ≤ 10−3.
References
- R. Fletcher and C.M. Reeves, Function minimization by conjugate gradients, Computer Journal, 7 (1964), 149-154. https://doi.org/10.1093/comjnl/7.2.149
- E. Polak and G. Ribiere, Note sur la convergence de directions conjugees, Rev Francaise Informat Recherche Operatinelle, 3e Annee. 16 (1969), 35-43.
- B.T. Polyak, The conjugate gradient method in extreme problems, USSR Computational Mathematics and Mathematical Physics, 9 (1969), 94-112. https://doi.org/10.1016/0041-5553(69)90035-4
- M.R. Hestenes and E.L. Stiefel, Methods of conjugate gradients for solving linear systems, Journal of Research of the National Bureau of Standards, 49 (1952), 409-436. https://doi.org/10.6028/jres.049.044
- Y.H. Dai and Y. Yuan, A nonlinear conjugate gradient method with a strong global convergence property, SIAM Journal on Optimization, 10 (1999), 177-182. https://doi.org/10.1137/S1052623497318992
- J.C. Gilbert and J. Nocedal, Global convergence properties of conjugate gradient methods for optimization, SIAM Journal on Optimization, 2 (1992), 21-42. https://doi.org/10.1137/0802003
- N. Andrei, Accelerated hybrid conjugate gradient algorithm with modified secant condition for unconstrained optimization, Numerical Algorithms, 54 (2010), 23-46. https://doi.org/10.1007/s11075-009-9321-0
- S. Babaie-Kafaki, R. Ghanbari and N. Mahdavi-Amiri, Two new conjugate gradient methods based on modified secant equations, Journal of Computational and Applied Mathematics, 234 (2010), 1374-1386. https://doi.org/10.1016/j.cam.2010.01.052
- Y.H. Dai and Y. Yuan, An efficient hybrid conjugate gradient method for unconstrained optimization, Annals of Operations Research, 103 (2001), 33 -47. https://doi.org/10.1023/A:1012930416777
- G. Li, C. Tang and Z. Wei, New conjugacy condition and related new conjugate gradient methods for unconstrained optimization, Journal of Computational and Applied Mathematics, 202 (2007), 523-539. https://doi.org/10.1016/j.cam.2006.03.005
- X.J. Zhang and A.N. Xu, Global convergence properties of a new class of nonlinear conjugate gradient methods, Guangxi Sciences, 12 (2005), 282-283.
- C.Y. Wang, S.Q. Du and Y.Y. Chen, Global covergence properties of three-term conjugate gradient method with new-type line search, Journal of Systems Science and Complexity, 17 (2004), 412-420.
- I. Bongartz, A.R. Conn, N. Gould and P.L. Toint, CUTE: constrained and unconstrained testing environment, Acm Transactions on Mathematical Software, 50 (1995), 123-160. https://doi.org/10.1145/200979.201043