A NONLINEAR CONJUGATE GRADIENT METHOD AND ITS GLOBAL CONVERGENCE ANALYSIS

Abstract

In this paper, we develop a new hybridization conjugate gradient method for solving the unconstrained optimization problem. Under mild assumptions, we get the sufficient descent property of the given method. The global convergence of the given method is also presented under the Wolfe-type line search and the general Wolfe line search. The numerical results show that the method is also efficient.

1. Introduction

We consider the unconstrained optimization problem

where f . Rn → R is a continuously differentiable function. The nonlinear conjugate gradient method is very useful for solving (1.1), especially when n is large. For any given x ∈ Rn, the nonlinear conjugate gradient method generates xk, k = 1, 2,...,n, by the following recursive relation

where gk = ∇f(xk) is the gradient of f at xk and βk is typically given by some formulas (such as [1-5]).

To achieve good computational performance and maintain the attractive feature of strong global convergence, in the past years, there exist many hybridizations of the basic conjugate gradient methods (see [6-10]). Based on the above papers, in this paper, we present a new hybridization nonlinear conjugate gradient method, where βk is given as

αk is computed by the Wolfe-type line search, which is proposed in [11]

where 0 <, δ <, σ <, 1, 0 <, γ <, 1. Based on the hybridization of βk as given by (1.4) we give the nonlinear conjugate gradient methods under the Wolfe-type line search and the general Wolfe line search.

In Section 2, we give the Method 2.1 and prove the global convergence of the proposed method with Wolfe-type line search. In Section 3, some discussions and the numerical results of the Method 2.1 are also given.

 

2. Method 2.1 and its global convergence analysis

Now, we give the Method 2.1 for solving (1.1).

Method 2.1

Step 1. Choose initial point x0 ∈ Rn, ε ≥ 0, 0 <, δ <, σ <, 1, u, γ ∈ (0, 1).

Step 2. Set d1 = −g1, k = 1, if ║g1║ = 0, then stop.

Step 3. Let xk+1 = xk + αkdk, compute αk by (1.5) and (1.6).

Step 4. Compute gk+1, if ║gk+1║ ≤ ε, then stop. Otherwise, go to next step.

Step 5. Compute βk+1 by (1.4) and generate dk+1 by (1.3).

Step 6. Set k = k + 1, go to step 3.

In order to establish the global convergence of the Method 2.1, we need the following assumption, which are often used in the literature to analyze the global convergence of nonlinear conjugate gradient methods.

Assumption 2.2

(i) The level set Ω = {x ∈ Rn|f(x) ≤ f(x1)} is bounded, i.e., there exists a positive constant C such that ║x║ ≤ C, for all x ∈ Ω.

(ii) In some neighborhood Ω of L, f is continuously differentiable and its gradient is Lipchitz continuous, i.e., there exists a constant L > 0, such that

for all x,y ∈ Ω.

Theorem 2.1. Let the sequences {xk} and {dk} be generated by the method (1.2), (1.3), and βk is computed by (1.4). Then, we have

for all k ≥ 1, where u ∈ (0, 1).

Proof. If k = 1, from (1.3), we get

Then, we can easily conclude (2.1). If k ≥ 2, multiplying (1.3) by , from (1.4), we get

Theorem 2.2. Suppose that Assumption 2.2 holds. By the Method 2.1, we have

Proof. From Theorem 2.1 and Assumption (i), we can know that {f(xk)} is bounded and monotonically decreasing, i.e., {f(xk)}, k = 1, 2,...,n, is convergent series. By (1.6), we have that

From Assumption 2.2, we get

So, from (2.3) and (2.4), we have

Square both sides of (2.5), we have

Therefore, by , we get

According to the convergence of {f(xk)}, we can conclude that

Remark 2.3. Suppose that Assumption 2.2 holds. By the Method 2.1, we know that

Proof. From Theorem 2.1, we know that

where u ∈ (0, 1).

Square both sides of (2.7), we have

Divided both sides of the above inequation by ║dk║2 , we get

From Theorem 2.2, we can conclude that

Theorem 2.4. Suppose that Assumption 2.2 holds. If {xk} (k = 1, 2,...,n) is generated by Method 2.1, we have

Proof. If (2.8) does not hold, there exists , such that

holds for all k ≥ 1. From (1.4), if , we have

From (1.3) and (1.4), we know that

Square both sides of (2.10) , we get

Divided both sides of the above equation by , we get

By

we have

By (2.9) and (2.11), we know that

Therefore, by , we have

If , we get

We can easily conclude that

which leads to a contradiction with (2.2). This shows (2.8) holds. We finish the proof of the theorem. □

 

3. Discussions of the Method 2.1 and Numerical Results

The line search in Method 2.1 can also given by the general Wolfe line search

where 0 <, σ1 ≤ σ2 <, 1.

Theorem 3.1. Suppose that Assumption 2.2 holds. Consider the Method 2.1, where αk satisfies (3.1), (3.2). Then, we have

Proof. From (3.2), we get

So

From (3.1), we have

By (3.3), we get

That is

We have

Discussion 3.1 By Theorem 3.1, we also can get the global convergence of the Method 2.1 with (3.1), (3.2).

Discussion 3.2 If the line search in the Method 2.1 is given by the other Wolfe-type line search, which is given in [12]

the method is also globally convergent.

Discussion 3.3 If in the Method 2.1, βk is given as

αk satisfies (1.5), (1.6) or (3.1), (3.2), we also can get the global convergence of the Method 2.1.

Numerical Results 3.4

Now, we test the Method 2.1, where αk satisfing (1.5), (1.6) or (3.1), (3.2) by using double precision versions of the unconstrained optimization problems in the CUTE library [13].

For the Method 2.1, αk is computed by (1.5) and (1.6) with δ = 0.4 and σ = 0.7 in the Table 3.1. αk is computed by (3.1) and (3.2) with σ1 = 0.5 and σ2 = 0.6 in the Table 3.2.

Table 3.1

Table 3.2

The numerical results are given in the form of NI/NF/NG/g, where NI, NF, NG denote the numbers of iterations, function evaluations, and gradient evaluations and g denotes the finally gradient norm. Finally, all attempts to solve the test problems were limited to reaching maximum of achieving a solution with ║gk║ ≤ 10−3.