Introduction
Building a testbed on Amazon EC2 has a number of benefits; the main one is being able to scale your tests out to the number of machines that Amazon will rent to you. Likely they are willing and able to rent more hardware than most are able to afford.
The downside to EC2 is that you don't have access to the underlying hardware, and thus you can't rely on an over-provisioned, low latency core of machines with multiple, high-bandwidth nics. Existing emulation testbeds rely on such a core in order to emulate high-bandwidth (typically up to 1Gbps), high latency links between machines.
For example, to emulate a single 1Gbps link, a relatively powerful machine with 2 1Gbps nics is used; the machine essentially delays and drops packets between the two nics; shaping the packets as desired by the experimenter.
With EC2, the max per-VM nic speed is 250Mbps and there is only one nic per VM. Furthermore, there is currently no way to specify where your EC2 instances (VMs) should be run, so every instance you create could be anywhere on Amazon's infrastructure. This makes determining things like bandwidth and latency between any two instances difficult. A final obstacle is the lack of precision timing interrupt support in the Xen kernel VM used EC2; the highest interrupt granularity is currently 10ms; which is too coarse for the purposes of packet shaping.
Clearly, emulating 1Gbps links in this environment is difficult.
Determining EC2 instance environment
Each time CloudLab runs an experiment, it needs to determine the network environment it has been deployed in. Things like: the latency, bandwidth and jitter between each of the instances that participate in the experiment. In order to do so a script is embedded in the EC2 instances, containing code that will run during instances startup performing a set of network measurements that gets reported back to CloudLab. These measurements then gets taken into account by shaping nodes.
Shaping packets
Whenever a packet that should be shaped gets transmitted, CloudHook, a linux kernel module intercepts the packet and stores it locally. CloudHook is a Netfilter module. Every few milliseconds a summary of what packets have been intercepted and when gets transmitted to the shaper node(s) responsible for shaping traffic on a given "virtual link".
The shaping node collects all the summaries of packets transmitted, and applies RED (Random Early Detection) on the packets. If a packet makes it through the link (doesn't get dropped), the summary gets sent back to the sender. The sender then contacts the receiver and transmits the packet to its destination.
At the receiver, the packet gets intercepted and stored by CloudHook. It then contacts a timing node notifying it that the packet has been received. The timing node sits in a tight loop (thus avoiding the timer interrupt issue) and will send a "deliver" instruction to Cloudhook on the receiver just at the time when the packet should be delivered. CloudHook receives the deliver instruction and delivers the packet to the destination network stack.
Discussion
What this approach buys you over intercepting and sending full packets to dedicated shaping nodes, is that you can emulate higher bandwidth links than any of the physical nics you have available. You can have multiple shaper nodes collaborating to shape one 40Gbps link, even though the nics themselves are much slower.
However, this approach will only work if the high performance shared links to be emulated are also relatively high latency. How low latency we can tolerate is something I'm currently working on.
Comments, outraged objections, etc to: einar@cs.cornell.edu. Last update: 26th November 2007