Dynamic Optimization of Active Queue Management Routers to Improve Queue Stability

Abstract

This paper aims to introduce the numerical methods for solving the optimal control theory to model bufferbloat problem. Mathematical tools are useful to provide insight for system engineers and users to understand better about what we are facing right now while experiment in a large-scale testbed can encourage us to implement in realistic scenario. In this paper, we introduce a survey of the numerical methods for solving the optimal control problem. We propose the dynamic optimization sweeping algorithm for optimal control of the active queue management. Simulation results in network simulator ns2 demonstrate that our proposed algorithm can obtain the stability faster than the others while still maintain a short queue length (≈10 packets) and low delay experience for arriving packets (0.4 seconds).

1. INTRODUCTION

The domination of the Internet by transport control protocol TCP-based services has drew many efforts to provide high network utilization with low loss and delay in a simple and scalable manner. Active queue management (AQM) algorithms attempt to achieve these goals by regulating queues at bottleneck links to provide useful feedback to TCP sources. While many AQM algorithms have been proposed and deployed, such as DropTail [1] and RED [2], most of them suffer from instability problem of queue [3]. In fact, a queue with small buffer size, overflow occurs frequently and dropping rate increases. On the other hand, a queue with large buffer size results into long end-to-end delay while throughput is saturated, i.e., TCP/AQM performance degradation problem. To stabilize queue performance, mathematical methods such as control theory [2] or dynamic optimization [4,5,6] can be applied to derive a suitable AQM algorithm with the purpose of maintaining the stability of queuing system in network routers.

As a frontier, control theory has been exploited for a long time in AQM research community. The basis is the fluid-flow nonlinear model proposed by Hollot et al. [2]. This model has two main equations describes the interaction between TCP Reno protocol and an AQM algorithm. Queue length is calculated at each time step by getting difference between the queue length value (at time t) and the desired queue length value, and controlling according to that difference. Control algorithms, such as proportional integral (PI) [2], cascade probability control (CPC) [7] have been applied to improve the quality of the dynamic TCP/AQM system. They, however, address only the subset of the stability problem. For example, PI controller tries to regulate queue length close to a constant value without considering of throughput stability.

The second class of solutions is numerical methods for solving optimal control of TCP/AQM model. There are many numerical methods able to solve the optimal control problems. The traditional approach to solving the optimal control problem entails first forming the optimality conditions, using the calculus of variations and Pontryagin's minimum principle, and then solving the resulting boundary value problem (BVP). The most popular methods for solving BVP are multiple shooting and collocation methods. This is known as the indirect approach for solving the optimal control problem. The references present just a small sample of the work that discusses or applies indirect approaches for the solution of optimal control problems [4,5,6]. Alternatively, one can a priori discretize the governing ordinary differential equations and the integral terms in the constraint functions or objective functional and thereby replace the infinite dimensional optimal control problem with a large nonlinear optimization problem. This is known as the direct method for solving the optimal control problems.

Our main contributions can be summarized as following:

• We develop a dynamic optimization sweeping algorithm based on the calculus of variations and Pontryagin's minimum principle for handling these kind of active queue management problem, briefly we call it as DO-AQM algorithm. The primary advantages of DO-AQM algorithm are easy to implement and no need to the initial guess for the adjoint functions (Section IV).

• We provide a simulation results of the proposed scheme for optimal control problem of TCP/AQM model (Section V).

The remainder of this paper is organized as follow: We will give a brief survey on the numerical methods for solving the optimal control problem in section II. The pros and cons for these methods are given as well. In section III, we present the direct approach for solving optimal control of TCP/AQM. Section IV describes the DO-AQM algorithm for solving our TCP/AQM model. Section V gives our simulation results of the proposed scheme and section VI finally concludes the paper.

 

2. RELATED WORK

Numerical methods for solving optimal control of TCP/AQM problems fall into two general categories as depicted in Fig. 1: indirect methods and direct methods. In an indirect method, first order necessary conditions for optimality are derived from the optimal control via the calculus of variations. These necessary conditions form a Hamiltonian boundary-value problem (BVP), which is then solved for extermal trajectories. The optimal solution is the found by choosing the extermal trajectory with lowest cost. In rare cases the solution can be obtained in closed form from the optimality conditions, but in general of these problems generally take the form of:

Fig. 1.Numerical Methods for OCPs.

• Differential algebraic equations (DAE) with boundary conditions.

• Parametric nonlinear optimization problem (Parametric NOP). According to the Pontryagin's minimum principle, the Hamiltonian function must optimized by the control variables at all points along the solution trajectory. For solving the Parametric NOP, one way is using the path-following method (PAFO). The other direction for solving the Parametric NOP is to discrete the problem then use the first or the second order sweeping methods based on the linearization or riccati equations, respectively (see Fig. 1). The primary advantages of indirect methods are their high accuracy in the solution and assurance that the solution satisfies the first-order optimality conditions. However, indirect approach have several disadvantages, including small radius of convergence, the need to analytically derive the BVP, an initial guess for the costate (adjoint), and if the path constraints are present, a priori knowledge of the constrained and unconstrained arcs. In a direct method, the continuous-time optimal control problem is converted to a nonlinear optimization problem (NOP) and the resulting NOP can be solved numerically using well-known software, for instant:

• L-Rambo, which is a solver for NOP with equality and inequality constraints. L-Rambo uses a total quasi Newton approach with line search and a working set strategy.

• Genetic algorithm solver (GAs) as Galib [8]. By using the GAs, there is no need to calculate the derivatives. However, the cost functional does not reach to the local solution as it claims in finding the global solution [4].

Although direct approach do not suffer from the disadvantages of the indirect method, many provide either an inaccurate costate or no costate information whatsoever. Fig.1 summarizes the numerical methods for OCPs. For more details on thess methods, we refer the reader to [4,5,6].

 

3. SYSTEM MODEL

3.1 Problem Formulation

We consider a network with a set of L of links and let Cl be the capacity of link l, for l = 1,....,|L|. Let a route r be a non-empty subset of L, and let R be the set of possible routes. Let S be a |R|×|L| matrix, set Srl = 1 if l ∈ r ; so that route r traverses link l and set Srl = 0 otherwise. Let route r generates at rate xr, the dynamical systems can be written as:

Where, the state of system Y(xr(t), wr(t)) can be described by the input rate of each user and the queue length at each router. Therefore, the ODE system (1-2) has the following state variables at time t : xr(t) for r = 1,....,|R|, and wl(t) for l = 1,...,|L|, where wl(t) denotes the queue length of router l at time t. Here, the control variable is defined as the dropping probability of router l at time l and is denoted by pl(t), for l = 1,...,|L|. To stabilize queue lengths, we define the following functional objective for ODE (1-2) as:

where w* is a constant denoting the desirable queue length. For more details on this model, we refer the reader to [9].

3.2 Direct Approach

In a direct method, the continuous-time optimal control problem (1-3) is converted to a nonlinear optimization problem (NOP) as:

subject to

where, and

The resulting NOP (4-6) can be solved numerically then using well-known software[8], which attempt to satisfy a set of conditions, called Karush-Kuhn-Tucker (KKT) conditions:

where, L(x, w, p) is the Lagrangian function defined as:

Ft (xt,wt,wt) = (w(t) -w*)T (w(t)-w*), the symbol T denotes the transpose operation and λ, μ are the adjoint variables.

 

4. DO-AQM ALGORITHM

To derive the DO-AQM, we firstly form the optimality conditions for problem (1-3), using the calculus of variations and Pontryagin's minimum principle and then solving the resulting boundary value problem. By defining the Hamiltonian function as:

where λ, μ are the adjoint functions. The Euler Lagrange equations are given by:

and the adjoint equations as:

and the stationarity condition is:

This leads to the following DO-AQM algorithm:

The DO-AQM algorithm is shown in Table 1. The first line shows the initialization for the control p0, the tolerance ϵ, and the maximum number of iterations MAXITER. Line 3 shows the initialization for the state functions xt1,wt1. It is obvious that at each iteration k of DO-AQM Algorithm consists of two sweeps, the original sweep (see the line 4 of Table 1) and backward sweep (see the line 11), through the time interval [t 0,tf] which one should integrate forward and backward, respectively, in time to obtain the state and the adjoint functions (see lines: 9, 10, 13, 14). In case k > 0, adjust the piecewise-constant control function by:

where, δpk is the step size from, see lines 5, 6, and 7. In line 16, we notice also that the algorithm terminates if the norm of the gradient of the Hamiltonian w. r. t. the control p, during the run time of the program is smaller than the tolerance ϵ or the maximum number of iterations has been reached.

Table 1.DO-AQM Algorithm

 

5. PERFORMANCE EVALUATION

5.1 Setup

We develop a simulation model to verify the performance of the proposed DO-AQM scheme using the packet-level network simulator ns-2 [10]. The chosen topology in Fig. 2 is a multi-bottleneck network which has been recommend to evaluating active queue management schemes [7,11,12]. There are totally n senders, specifically we pick n = 5, 3, switches and one receiver. Each sender continuously sends FTP data using TCP Reno transport protocol to the receiver. We conduct three tests for performance comparison purpose between our proposed DO-AQM algorithm and the two current deployed schemes, e.g, DropTail and Adaptive-RED (ARED). The simulation time in ns-2 is measured in unit of ticks, i.e, each tick is approximately equal to 0.1 seconds. The results are then written into ns-2 trace files format and further processed using awk script language.

Fig. 2.Simulated network topology with multi-bottlenecks.

5.2 Results and compared with other approaches

In this section, we analyze our produced results from simulation ns2 and compare DO-AQM with other approaches of current deployed AQMs. At the first glance, from Fig. 3, we can see our proposal achieves the lowest queue length value. Through time, the queue length evolves as saw-tooth shapes. These saw-tooth shapes imply that our proposed model works correctly because we assume additive increase multiplicative decrease (AIMD) in our model in the section III. With the lowest queue length value of DO-AQM, the performance of queue inside router is the best because the bottleneck links are not busy all time. Therefore, new arrival packets will be accepted by transport protocols (in this simulation example, we use TCP Reno as transport protocols).

Fig. 3.Queue length in packets at bottlenecks.

Secondly. the delay comparison results are presented in Fig. 4. We can easily realize that there is a similarity between queue length and delay results of all AQMs which are the saw-tooth shapes. In case of our proposed DO-AQM in Fig. 4(c), we can see that delay peak of DO-AQM is approximately 0.42 (sec), while DropTail sometimes results into large delay of 1.1 (sec). These interesting results confirm that DropTail has a problem of long waiting time in queue for every accepted packets which motivates us to consider dynamic optimization active queue management implementation.

Fig. 4.Delay comparison for different AQM schemes.

Finally, we show the end-to-end throughput comparison for different active queue management schemes (Fig. 5). There are two kinds of throughputs in networking area. The first one is indiscriminate throughput (in short, throughput) which is the average amount data received by the receiver per unit time, regardless of whether the data is re-transmission or not. The second one is good throughput (in short, goodput) which is the average amount data received by the receiver per unit time that are not re-transmitted. In the scope of this article, we focus on indiscriminate throughput only to evaluate user-experience. Looking into our result in Fig. 5(c), we can see that DO-AQM brings a good stability in throughput values. While DropTail and ARED's throughput are oscillated so much (from 0 to 0.3 Mbps), our proposed scheme DO-AQM is more stable, i.e. the throughput value is small oscillated from 0.02 to 0.18 Mbps.

Fig. 5.Throughput comparison for different AQM schemes.

 

6. CONCLUSION

There are several approaches to handle the optimal control problem. We have proposed a dynamic optimization sweeping methods for solving the optimal control of AQM model. Although the DOAQM approach needs to derive the adjoint equations which is not easy for some practical problems but it is much faster and more accurate than the direct approach. Simulation results in network simulator ns2 demonstrate that our proposed algorithm are also discussed.