CS6410 Fall 2013

TR 10:10-11:25AM

Phillips Hall 407

 CS6410: Advanced Systems


[Lab 0 Part B] Working with Xen-Blanket

If you have not finished [Lab 0 Part A], please finish it first. This part of the lab requires the knowledge and materials from the first part. You should use your own security group, keypairs, and so on from Part A, which are indicated as gp-netid-xxx, kp-netid-xxx, or id-rsa-kp-netid-xxx in this webpage.

In this lab you will install and run Xen-Blanket in Amazon EC2 and bundle the image. Xen-Blanket is a virtual machine monitor (VMM) that runs on a VMM. Thus, running a Xen-Blanket on Amazon EC2 is nesting two virtualized layers in the cloud. For more information about Xen-Blanket, visit http://xcloud.cs.cornell.edu.

1. Building and Installing Xen-Blanket in EC2

These instructions are for running the Xen Blanket on an Amazon EC2 cluster instance. The key components are a patched Xen and blanket drivers implemented as Linux modules.

  1. Before you start an Amazon EC2 instance, please mark that you are using the instance in the shared document. The link to the document can be found in CMS. We have limited shared budget to access EC2, so we don't want any instance to be running when you are not actually doing something on it. Instances running without the usage indication in the shared document can be terminated without prior notice.
  2. Launch Amazon EC2 instance EC2 CentOS 5.4 HVM AMI (ami-7ea24a17). The instance must be set to cluster compute instance (using option -t cc1.4xlarge).
    aws run-instances ami-7ea24a17 -g gp-netid-xxx -k kp-netid-xxx -t cc1.4xlarge
  3. Log into the Amazon instance using a ssh client. You may have to wait for a while for the instance to be completely booted.
    ssh -i ~/.aws/id-rsa-kp-netid-xxx root@ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com
  4. Download and unpack the Xen-Blanket patches (xen.blanket.1.0.1.tar.gz), then enter the release directory.
    wget http://xen-blanket.googlecode.com/files/xen.blanket.1.0.1.tar.gz
    tar zfx xen.blanket.1.0.1.tar.gz
    cd xen.blanket.1.0.1
  5. We must next obtain the third party Xen and Linux sources that we will apply the Xen-Blanket patches against. They are from the CentOS distribution; the original source RPM (kernel-2.6.18-194.26.1.el5.src.rpm) can be found in the CentOS vault at http://vault.centos.org/5.5/updates/SRPMS/. For convenience, we have already assembled the source RPM into a tarball, available at http://xcloud.cs.cornell.edu. From the release directory, download the source tarball, then run the create_source_tree script. The create_source_tree script patches the supplied kernel and Xen, and also copies and patches PV-on-HVM drivers and virtio drivers to make them work as blanket drivers.
    wget http://xcloud.cs.cornell.edu/code/kernel-2.6.18.tar.gz
  6. We have now patched the Linux and Xen source to include support for blanket drivers. Next, we must install the Xen tools, build and install the blanket-enabled Xen, and build and install blanket driver modules. The modules can be built against a ready-built Linux kernel or we can build a kernel from scratch. Yum can be used to avoid building most things from scratch. To be sure that packages obtained from yum are compatible with the patched code version, we first change the yum repositories to point to an older repository by copying the included .repo file. The following yum line installs the Xen tools, a xen-enabled Linux kernel for domain 0, and Xen. In the next step, we will be replacing the installed Xen with one that supports blanket drivers.
    If you are using xen-blanket version 1.0.0, there is a minor error to fix in the CentOS-Base-Blanket.repo file. You should open the file with a text editor and remove the first line that says,
    [root@ip-10-17-185-112 kernel-2.6.18]# cat /etc/yum.repos.d/CentOS-Base.repo.
    Now you are ready to proceed. Execute the following commands.
    cp CentOS-Base-Blanket.repo /etc/yum.repos.d/CentOS-Base.repo
    yum clean all
    yum install -y xen xen-devel xen-libs xen-libs-devel kernel-xen kernel-xen-devel gcc
  7. Make sure the PV drivers will be loaded for the network and the disk on boot by putting aliases in the /etc/modprobe.conf file. The network alias should already be there.
    #alias scsi_hostadapter ata_piix
    alias scsi_hostadapter xen-vbd
    alias eth0 xen-vnif
  8. We must still build and install the blanket-enabled Xen and the blanket driver modules for Linux. First we enter the patched source directory.
    cd kernel-2.6.18
    By default, the Makefile is set up to build a Linux kernel from scratch. To avoid this, edit the Makefile to build the blanket drivers against the second-layer domain 0 Linux kernel installed two steps before. For example, to find out the name of the kernel sources, and the kernel version for modules run:
    # ls /usr/src/kernels/|grep el5xen
    # ls /lib/modules/ |grep xen
    and then change the KERNEL_SRC and KERNEL_VERSION lines in the Makefile to match:
  9. Run make to build and install Xen and the blanket drivers for Xen (Amazon is Xen based):
    make xen_install
    make blanket_drivers_xen_install
  10. Replace the /boot/grub/menu.lst with the code below. It sets the default to the Xen entry, and replace the xen line with the newly installed Xen. It replaces the linux images to load when system reboots. Make sure the kernel and module names names match the ones in your /boot folder.
    serial --unit --speed=57600
    terminal --timeout=1 console serial

    title CentOS (2.6.18-238.9.1.el5xen)
    root (hd0,0)
    kernel /xen-3.1.2-blanket.gz com1=115200,8n1 console=com1
    module /vmlinuz-2.6.18-194.32.1.el5xen ro root=/dev/VolGroup00/LogVol00 rhgb quiet console=hvc0 earlyprintk=xen ide0=noprobe
    module /initrd-2.6.18-194.32.1.el5xen.img
  11. Change the default network setup behavior of Xen away from bridging, in /etc/xen/xend-config.sxp. NAT is one option. Comment out the following lines using #.
    #(network-script network-bridge)
    #(vif-script vif-bridge)
    Add the following new lines.
    (network-script network-nat)
    (vif-script vif-nat)
  12. You are almost done. Before you reboot check your linux version.
    cat /proc/version

    Linux version 2.6.18-164.15.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Wed Mar 17 11:30:06 EDT 2010
    Copy the information or remember the linux version so that you can compare it with the version that you will see once the xen-blanket starts to run.
  13. Now Reboot! When the instance comes back up, you should be able to log into the second-layer domain 0. Be patient. Rebooting takes quite a while. If you can connect and log in, the blanket drivers are working. Check the linux version using the same command from the previous step. You will see a different version number.
    Linux version 2.6.18-194.32.1.el5xen (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Jan 5 18:44:24 EST 2011

2. Bundling Xen-Blanket Image as EBS Image

Once you have verified that you are done with the previous steps, you should bundle your image. Different from Lab 0 Part A, you should use different commands to bundle the image, because the image size of cc1.4xlarge instance exceeds the standard size (10GB). The image must be saved as an elastic block storage (EBS) image, which supports up to 1TB. To save your instance as an EBS image, you should first figure out your instance id using command while your instance is running (Note that "prompt>" is for the commands typed in your local machine, not in the instance in Amazon EC2. Replace netid with your own netid..)
prompt> aws describe-instances | grep netid
The instance id is the first returned value in the format of i-xxxxxx. Now you can bundle your image using the following command with the instance id. You should leave the instance to be running at this point.
prompt> ec2-create-image -n "im-netid-xen-blanket" i-xxxxxx
This command will immediately return an image id (ami-xxxxxxx), but you won't be able to connect to your instance for a while. Wait for several minutes then you will be able to connect back to instance using ssh. If you want to check whether your image is ready you can use the command,
prompt> ec2-describe-images -o self |grep ami-xxxxxxx
This will return whether your image is available or pending. When the state becomes available, your image is ready for use. For this image you don't have to modify the manifest file as you did in Lab 0 Part A.

3. Terminate Instance

As the final step DO NOT FORGET TO TERMINATE THE INSTANCE. Please understand that the large EC2 instance that you are using for this part costs much more than the one used in part A.
aws terminate-instances i-xxxxxx
Also unmark the usage indication in the shared document.

4. End of Lab 0

At this point you are done with Lab 0! Double check that all your instances are terminated, and two ami images from both Part A and B are properly created in EC2.

Most of the instructions in this page came from http://xcloud.cs.cornell.edu/code/README.txt