Figure 1.1 Testbed Setting
While providing differentiated Internet services becomes one of the hottest topics in current network research, most of the work focuses on the architectures (e.g. differentiated services) and routing mechanisms (e.g. QoS routing) in the range of WAN. Little interest has been shown on studying its implication to LANs, especially the legacy Ethernet which don't provide any QoS support, though the stub networks are the first and the final steps of an "Internet path" to meet the QoS requirements. Usually the argument about this is either the Ethernet are currently lightly loaded, or we can easily over-dimension them to make them so. The questions which follow naturally are "To what extent do we need to over-dimension them to meet the applications' requirements?"
The current focus of this work is to examine how the (high) load situation of Ethernet affects the dynamics and performance of several popular network applications. In particular, the two applications which we feel interested in are web access and internet telephony. The former is a very popular network application which greatly contributes to the current internet traffic. Its traffic pattern is characterized by small client requests one way and bulk data transfer as server responses the other. The latter has been proposed as a highly desirable application in lieu of the traditional telephone infrastructure and now is only at its experimental stage. Its traffic is featured by a series of evenly spaced, small size audio packets. Due to the different traffic characteristic of and transportation protocols used by these two applications, the impacts of the increasing Ethernet load on them are quite different. We are interested in studying what these impacts are, especially, when the degraded performance fails to meet the (basic) application requirements. We are also interested in improving the implementations of these applications to accommodate increasing load on the Ethernet.
We use simulation to study the problem. To avoid the complication when WAN technology is involved, we restrict the problem to a simple case when all the participants of the applications sit on the same LAN.
To study the impact of Ethernet load on applications, we need two components in our simulation: the behavior of the participants (servers and clients in web application, participants in Internet telephony), the LAN to which all the participants are connected. We decompose the simulation by using a simulator for LAN and using real application or simulator for participants.
We use Entrapid on top of Jia Wang's efficient Ethernet simulation[1] to simulate the LAN. We choose Entrapid for three reasons. Firstly, from a developer's perspective, Enrapid provides the abstraction of ``a network in a box''. It supports multiple Virtualized Networking Kernels (VNKs). Each VNK corresponds to a machine on the Internet, and each virtualized process corresponds to a process running on that machine. Since it implements TCP/UDP stack, it is actually an emulator and suitable for studying TCP/UDP behavior. Second, it supports RealNet technology which can seamlessly connect real world devices such as routers and switches to the emulated network. So we can connect our participants (simulation or real application) to the ``Ethernet box'' simulated by Entrapid in the normal way they connect to real Ethernet and test the behavior of the participants when interacting with the network protocols already implemented within Entrapid. Finally, the efficient Ethernet simulation adopted by Entrapid facilitates adjusting the load situation of the Ethernet. Instead of using time-consuming CSMA/CD simulation, Entrapid now has a built-in mapping between the carried load and an empirical delay distribution. By monitoring the carried load, Entrapid can give an accurate estimation of the packet delay by generating a randomized delay value according to the distribution. In addition, by providing a way to directly set the background offered load and average packet size , we can get the effect of heavy cross-traffic for free, while in traditional Ethernet simulation, the same amount of load has to be generated packet by packet.
Figure 1 shows the simulation setting.
Here we denote the ethernet interfaces by the IP addresses associated with them. Three
machines were set up with Entrapid running on the middle one, and user applications
running on the other two. Two virtual machines, m0 and m1,
are created in Entrapid to simulate the LAN. Using RealNet technique, we can bind each
virtual machine to a real network interface (m0 to interface 172.16.4.1,
m1 to interface 172.16.5.1 in Figure 1). By setting the routes, we can enforce all packets
exchanged between the client and the server to go through the virtual link between m0 and
m1. These packets are subject to the delay determined by traffic characteristic on LAN,
which is a combination of the background offered traffic and the traffic generated by this
application. By changing the load of Ethernet in Entrapid, we can study its impact on the
application by observing the corresponding change of the participants' behavior.
Figure 1.1 Testbed Setting
We use Apache1.3.4 as our HTTP server, which is a popular free web server which incorporates many HTTP1.1 features. It also generates server-side log file for the purpose to evaluate server performance. To generate the Web traffic and simulate the behavior of web clients, we use Scalable URL Reference Generator (SURGE) from Boston University. SURGE generates a sequence of URL requests which exhibit representative distributions for document popularity, document sizes, request sizes, temporal locality, spatial locality, embedded object count and off times. It simulates different versions of HTTP: HTTP1.0, HTTP1.1 without pipelining, and HTTP1.1 with pipelining. Moreover, it creates client-side log file which facilitates analysis.
The performance metric that we focused on is latency, defined as the time between the sending of the client request and the completion of receiving the server response. Latency greatly depends on network conditions. As the average packet size and the Ethernet load vary, the packet delay and goodput (hence droprate) change accordingly. Latency also differs with the different design of web clients in different versions of HTTP. HTTP1.0 opens one connection for each file in each object (An object is defined as a base html file with zero or more embedded files). It requires at least two RTTs per document or inlined image hence causes unnecessary latency. It also incurs additional connection setup-maintain-teardown cost and the short connections suffer the TCP slow-start inefficiency. HTTP1.1 without pipelining improves version 1.0 in two ways:
1. allowing multiple connections to be opened and files
to be retrieved concurrently;
2. allowing multiple files to be retrieved via an open
connection, to alleviate the latency, additional cost and short connection problem.
Pipelining further improves the HTTP1.1 performance by requesting all the embedded files at one time by sending them along the connection, without waiting for the responses to any of the individual requests. This pipelining will further decrease the latency. We plan to investigate the impact of increasing Ethernet load on each of the client designs to compare their "resistance'' to pathological network situation.
As the delay and drop rate increase, the web traffic subjects to complicated TCP retransmission and congestion control. To study how this behavior affects latency, as the first stage work, we study the simplified case when there is only one client retrieving a base HTML file (HTML file with no imbedded objects) with fixed size using HTTP1.0. The methodology we adopted is to first study the relation between droprate and latency when eliminating the affect of delay (using a fixed delay 0 for each packet), and the relation between delay and latency when eliminating the affect of droprate (using a fixed droprate 0), then study the combined affect of droprate and delay on latency determined by offered load. Figure 2.1 and 2.2 give various offered load and their corresponding droprate and delay generated by Entrapid, which reflect the empirical data.
Figure 2.1 Load v.s. Droprate, generated by Entrapid
Figure 2.2 Load v.s. Delay, generated by Entrapid
From the simulation data, we can see that in general, mean latency increases linearly with the average packet delay (Figure 2.3), but exponentially with high drop rate ( drop rate > = 10% ) (Figure 2.4), and the impact of load on latency is the combination of these two (the exact "combination" need to be further studied) (Figure 2.5). As [2], we observed that the latency displays heavy tails and high variance, particularly for long transfers with high loss rates. [2] proposed an analytical model for estimating the median latency. For comparison, Figure 2.4 also shows the estimated median latency (the "estimated latency" curve).
Figure 2.3 Delay v.s. Latency for fixed
droprate = 0, one HTTP1.0 client retrieves one file with size 10,000 bytes
Figure 2.4 Droprate v.s. Latency for fixed delay = 0, one
HTTP1.0 client retrieves one file with size 10,000 bytes
Figure 2.5 Load v.s. Latency, one HTTP1.0 client retrieves
one file with size 10,000 bytes
We found there is a mismatch between the analytical model and the simulated median latency, when the loss rate is high (>= 8%). The analytical model failed to capture the superlinear increasing of the median latency shown by the data. We believe the mismatch comes from the problematic model of the TCP timeout. Since timeout is the primary contributor of the latency when delay is large and droprate is high, we expect the correct estimatioin of timeout should follow the same superlinear growing trend as the latency. But as we can see from the graph, this is not the case. The "TO coeff" curve displays the estimated timeout as the coefficient of initial timeout value T0 (which generally is some multiple of the estimation of RTT). Assume that the estimation of RTT is relatively accurate, since RTT is mainly decided by the delay added by Entrapid, which is a randomized value determined by empirical delay CDF and doesn't have a large variance, the timeout coefficient is approximately proportional to the timeout. The timeout coefficient curve (almost parallel to the estimated latency curve) exhibits a different trend from the simulated median latency. We are currently trying to improve the analytical model by checking the model for timeout. We believe that having a good model for median latency in high drop rate situation can help to clarify the interaction between TCP retransmission/congestion control and HTTP, on which all the work aiming at improving HTTP performance should be based.
3. The impact of Ethernet load on Internet telephony
We use the data exchange code of the Internet telephony application (provided by CS519 project group Phonedation team 1 [3]). The sender/receiver code implemented adaptive playout and Forward Error Correction (FEC) algorithm. We instrumented it for performance analysis.
The metrics directly affect user-observed quality are the number of packets missed (including truly missed packets and packets arriving too late to be played out), the number of packets played using primary encoding, the number of packets played using secondary encoding (i.e. the number of packets saved by FEC), and the playout delay. These metrics, in turn, are decided by the interaction between playout and FEC algorithm and network characteristics including delay, delay jitter and droprate. Delay jitter determines the playout (buffering) delay and size of playout buffer needed at the receiver side. Though delay is not so significant as delay jitter, it is bounded by the demand of real-time communication (Think about voice mail as an internet phone call with unusually large delay!). Droprate contributes significantly to the degraded quality, since adaptive playout can only avoid packet loss due to large delay jitter, truly lost packets may only be saved by FEC, provided their corresponding secondary-encoding packets are neither lost nor too late.
Our primitive simulation result shows that as the Ethernet load grows, both mean and variance of the packet delay (hence delay jitter) and droprate increase. With high droprate (>=10%), the quality degrades greatly because adaptive playout can't help and the ability of FEC is hampered by loss or increasing delay of the secondary-encoding packets. Also, the playout delay increases and eventually becomes too large to be tolerated, when the playout buffer are full of delayed packets and overflows. However, we hope this pathological case will occur as late as possible. We will further investigate the different parameters used in the algorithm, or improving the algorithm itself, to see if we can delay the pathological case and what is the highest load the algorithm can tolerate.
Acknowledgements
Sincere thanks to Vincent Ng, Kiri Wagstaff, Westley Weimer for providing the access to their Internet telephony data exchange source code. Thanks to Westley Weimer for providing the access to his Linux demo code and the right to play out his passionate poem recording. Finally, we thank Ensim Corporation for helping us to set up the testbed.
Reference
[1] Efficient and accurate
Ethernet simulation, J. Wang and S. Keshav, Cornell CS Technical Report,
Feburary 1999.
[2] Modeling the performance of short TCP connections, N. Cardwell, S. Savage, and T.
Anderson, Dept. of Computer Science and Engineering, University of Washington, October
1998.
[3] CS519
Phonedation team data exchange group final report.
Appendix
The presentation slides of Boom'99 are available here.