CS614: Advanced Course in Computer Systems

Fall 2007

Instructor: Ken Birman
4119B Upson, x5-9199
Ken can be found in his office daily, 10am-2pm except at lunch time 

TA: Lakshmi Ganesh, Office Hours: Thurs. 11:30 – 1:00, 304 Upson Hall

Classes: Tuesdays and Thursdays, 10:10-11:25; Upson 109

A page with details on the class is here. 


Who (click for slides)



Required Reading

Suggested Reading



Organizational Meeting


Background materials for one of the ideas for a possible project



Concurrency, Threads, and Events.

Using Threads in Interactive Systems.  Hauser et al.  14th SOSP, Dec 1993.

Goal-oriented programming, or composition using events,or threads considered harmful. Van Renesse. ACM SIGOPS European Workshop, Sep 1998.

Eraser.  A Dynamic Data Race Detector for Multi-Threaded Programs.  Savage et al.  16th SOSP, 1997.

SEDA: An Architecture for Well Conditioned, Scalable Internet Services.  Welsh et al. 18th SOSP, Oct 2001.



File Systems

Soft Updates: A Solution to the Metadata Update problem in File Systems.  Ganger et al. ACM TOCS 18(2). May 2000.

A Fast File System for UNIX.  McKusick et al.  ACM TOCS 2(3), Aug 1984.

The Zebra striped network file system; Hartman and Ousterhout; 14th SOSP, 1993.

When to forget in the Elephant file system.  Santry et al.  17th SOSP, 1999.

The Design and Implementation of a Log-Structured File System.  Rosenblum and Ousterhout.  13th SOSP, Oct 1991.



OS Kernels

The Performance of µ-Kernel-based Systems.  Härtig et al.  16th SOSP, Oct 1997.

The UNIX Time-Sharing SystemRichie and Thompson.

EMERALDS: A Small-Memory Real-Time Microkernel.  Zuberi et al.  17th SOSP, 1999.

The Flux OSKit: A Substrate for OS and Language Research. Ford et al. 16th SOSP, Oct 1997.





  .  Asbestos part)


  Virtual Memory

Machine-Independent Virtual Memory Management for Paged Uniprocessor and Multiprocessor Machines, Rashid et al.  IEEE TOC C-37(8), Aug 1988.

Virtual Memory Primitives for User Programs.  Appel and Li.  4th ASPLOS, April 1991.

Sharing and Protection in a Single Address Space Operating System Chase et al. ACM TOCS, November 1994.

Labels and Event Processes in the Asbestos Operating System. Petros Efstathopoulos et. al.  SOSP 2005



  Extensible Kernels

Exokernel: an operating system architecture for application-level resource managementEngler et al.  15th SOSP, Dec 1995.

Application performance and flexibility on Exokernel systems.  Kaashoek et al.  16th SOSP, 1997.

Microkernels meet Recursive Virtual Machines.  Ford et al.  2nd OSDI, Oct 1996.

Extensibility, Safety and Performance in the SPIN Operating System Bershad et al.  15th SOSP, Dec 1995.

   Matt Watkins



Disco: Running Commodity Operating Systems on Scalable Multiprocessors Bugnion et al.  16th SOSP, 1997.

Towards Transparent and Efficient Software Distributed Shared Memory.  Scales and Gharachorloo.  16th SOSP, 1997.

Performance Isolation: Sharing and Isolation in Shared-Memory Multiprocessors.  Verghese et al. 8th ASPLOS, Oct 1998.

Tornado: maximizing locality and concurrency in a shared memory multiprocessor operating system.  Gamsa et al. 3rd OSDI, Feb 1999.

   Jon Kaldor


  Virtual Machine Monitors: Technology and Trends

Xen and the Art of Virtualization, Dragovic, et. al.  Proc 19th SOSP, Oct. 2003

Memory resource management in VMware ESX server, C. A. Waldspurger.  OSDI 2002

Operating System Support for Virtual Machines, S. T. King, G. W. Dunlap, and P. M. Chen. 2003 USENIX Technical Conference, 

Virtual Machine Monitors: Current Technology and Future Trends, Mendel Rosenblum, Tal Garfinkel. Computer, vol. 38, no. 5, pp. 39-47, May 2005;

   Jonathan Winter


  If the CPU is so fast, why are the programs running so slowly?

Continuous Profiling: Where Have All the Cycles Gone? Jennifer M. Anderson et. al. ACM TOCS; 15:4 (Nov. 1997) 

  - The DEC Alpha 21164 Data Sheet ftp://ftp.digital.com/pub/Digital/info/semiconductor/literature/21164ds.pdf

-    An IEEE Micro article about the 21164http://ieeexplore.ieee.org/iel1/40/8521/00372349.pdf?tp=&arnumber=372349&isnumber=8521

Also, this website describes the performance counters available in a variety of microprocessors and provides a library that can be integrated into C, C++, Fortran, and Java programs to gather information from the performance counters. http://www.fz-juelich.de/zam/PCL/

Power Architecture: A High-Performance Architecture with a History.  IBM Technical White Paper.

Intel Pentium 4 Processor.  Intel Technical White Paper

System Support for Automated Profiling and Optimization.  Zhang, et. al.  SOSP 1997

   Xin Zheng


  Remote method invocation: making distributed computing totally transparent…

Implementing Remote Procedure Calls.  Birrell and Nelson.  ACM TOCS 2(1), Feb. 1984.

RPC in the x-Kernel: Evaluating New Design Techniques. Peterson et al.  12th SOSP, Nov. 1989.

Performance of Firefly RPCSchroeder and Burrows.  ACM TOCS 8(1), 1990.

   Kiyan Ahmadizadeh


  To infinity and beyond!

U-Net: A User-Level Network Interface for Parallel and Distributed  Computing.  Von Eicken, Basu, Buch and Werner Vogels. 15th SOSP, Dec1995,   

Evaluation of the Virtual Interface Architecture (VIA)

Lightweight remote procedure call; Bershad et al.  ACM TOCS  8(1), Feb 1990.

   Matt Watkins


  System Design

Hints for Computer Systems DesignLampson.  ACM OSR 15(5), Oct 1983.

The Impact of Architectural Trends on Operating System Performance.  Rosenblum et al.  15th SOSP, 1995.

Interposition Agents: Transparently Interposing User Code at the System Interface.  Jones.  14th SOSP, 1993.

End-to-End Arguments in System DesignSaltzer et al.  ACM TOCS 2(4), Nov 1984.

   Xin Zheng


  Network Objects

Network Objects.  Birrell et al.  14th SOSP, Dec 1993.

Fine-Grained Mobility in the Emerald System
Jul et al. ACM TOCS 6(1), Feb 1988.

The Jini architecture for network-centric computing. Waldo. CACM 42(7), Jul 1999.

Linda in Context.  Carriero and Gelernter.  CACM 32(4), Apr 1989.

  Jon Kaldor


  Stopping Worm/Virus Attacks

Vigilante: End-to-End Containment of Internet Worms. Manuel Costa et. al. SOSP 2005

The Taser Intrusion Recovery System. Ashvin Goel et. al. SOSP 2005

Scalability, Fidelity and Containment in the Potemkin Virtual Honeyfarm. Michael Vrable et. al. SOSP 2005

   Jonathan Winter



Rethink the Sync.  Edmund B. Nightingale, et. al.
OSDI '06.


Speculative Execution in a Distributed File System.  Edmund B. Nightingale, Peter M Chen, Jason Flinn.
SOSP 2005  

 Tuan Cao



The Information Bus---an Architecture for Extensible Distributed Systems. Oki et al. 14th SOSP, Dec. 1993.

Mesh-based Content Routing using XML.  Snoeren et al.  18th SOSP, Oct 2001.

Design and evaluation of a wide-area event notification service.  Carzaniga et al.  TOCS 19(3), Aug 2001.

Matching Events in a Content-based Subscription System.  Aguilera et al. 18th PODC, 1999.

  Yao Yue


  Staggeringly large file systems

File and storage systems: The Google file system. Ghemawat, Gobioff, Leung. SOSP-19, Oct 2003.  

A Distributed Decentralized Information Storage and Retrieval System.  Ian Clark.  University of Edinburgh, 1999 (Original unpublished technical report on Freenet).

Why Gnutella can't scale, no really?  Ritter, Feb 2001.

Storage management and caching in PAST, a large scale, persistent peer-to-peer storage utility.  Rowstron and Druschel.  18th SOSP, Oct 2001.

  Kiyan Ahmadizadeh


  Application-Level Multicast Routing

Enabling Conferencing Applications on the Internet using an Overlay Multicast Architecture, Chu et al. ACM SIGCOMM, Aug 2001.

SCRIBE: A large-scale and decentralized application-level multicast infrastructure.  Castro et al.  IEEE Journal on Selected Areas in communications (JSAC), 2002.

Overcast: Reliable Multicasting with an Overlay Network  Jannotti et al.  4th OSDI, Dec 2000.

  Yao Yue

  (Ken out of town, Lakshmi Ganesh will cover for him)


  Peer to peer dissemination

Some observations on BitTorrent performance.  Ashwin R. Bharambe, Cormac Herley, Venkata N. Padmanabhan.  SIGMETRICS 05.

Resilient overlay networks. David Andersen, Hari Balakrishnan, Frans Kaashoek, Robert Morris.  SOSP ‘01

SplitStream: high-bandwidth multicast in cooperative environments.  Castro, et. al.  SOSP 2003.


(“chalk talk”)


  Epidemic Techniques

Epidemic algorithms for replicated database maintenance; Demers et al.   6th PODC, 1987.

Kelips: Building an Efficient and Stable P2P DHT Through Increased Memory and Background Overhead. Indranil Gupta, Ken Birman, Prakash Linga, Al Demers and Robbert van Renesse.  2nd International Workshop on Peer-to-Peer Systems (IPTPS '03); February 20-21, 2003.  Claremont Hotel, Berkeley, CA, USA.

Bimodal Multicast Birman et al.  ACM TOCS 17(2), May 1999.

Astrolabe: A Robust and Scalable Technology for Distributed System Monitoring, Management, and Data Mining.  Van Renesse et al.  ACM TOCS 2003 (to appear).

  Barry Burton



  Ordering and Consistent Cuts

Time, Clocks, and the Ordering of Events in a Distributed System. Lamport. CACM 21(7). July 1978.


Distributed snapshots: Determining global states of distributed systems. Chandy, Lamport. ACM TOCS 3(1), 1985, 63-75.

  Lakshmi Ganesh




Optimal Clock Synchronization.  Srikanth and Toueg.  JACM 34(3), July 1987.

Understanding protocols for Byzantine Clock Synchronization. Schneider.  Cornell CS TR 87-859.  Aug 1987.

Using Time Instead of Timeout for Fault-Tolerant Distributed Systems. Lamport.  ACM TOPLAS 6:2, 1974

Probabilistic Internal Clock Synchronization.  Cristian and Fetzer.




Impossibility of Distributed Consensus with One Faulty Process. Fisher et al.  JACM 32(2), Apr 1985.

Revisiting the Paxos Algorithm  De Prisco et al. WDAG, Sep 1997.

The weakest failure detector for solving consensus; Chandra et al. J. ACM 43, 4, Jul. 1996.

Paxos Made Simple.  Lamport.  ACM SIGACT NEWS 32(4).  Dec. 2001.



  Virtual Synchrony

The Process Group Approach to Reliable Distributed Computing. Birman. CACM, Dec 1993, 36(12):37-53.

From Set Membership to Group Membership: A Separation of Concerns.  A. Schiper and S. Toueg.  IEEE TDSC 3:1 (Jan 2006)


A modular approach to fault-tolerant broadcast and related problems.  Hadzilacos and Toueg.  Cornell CS TR 94-1425,May 1994.

Building reliable, high-performance communication systems from components Liu et al.  17th SOSP, 1999.

Ken’s slide set on Scalable Trusted Computing (Nov. 2006).  This discusses the role of consensus and group communication in the context of an emerging problem, namely tracking a dynamically evolving security policy database.

Understanding the Limitations of Causally and Totally Ordered Communication.  Cheriton and Skeen.  14th SOSP, 1993.

  Note: This paper has some flaws.  Discussion and rebuttals to some of the claims it makes appeared in the January 1994 issue of Operating Systems Review



  Practical Replication

Dangers of Replication and a Solution.  Gray et al.  ACM SIGMOD, Jun 1996.

Maintaining Availability in Partitioned Replicated Databases. Abbadi and Toueg.  ACM TODS 14(2), Jun 1989.


Robbert and Fred: Chain Replication


The Costs and Limits of Availability for Replicated Services.  Yu and Vahdat.  18th SOSP, Oct 2001.

  Dan Freedman


  Game Theory

BAR Gossip.  H. Li, A. Clement, E. Wong, J. Napper, I. Roy, L. Alvisi, M.Dahlin, OSDI 2006, Nov 2006.  How Bad is Selfish Routing? T. Roughgarden and E. Tardos.  JACM, 2002, Volume 49 , Issue 2.

A BGP-based Mechanism for Lowest-Cost Routing.  J.  Feigenbaum, C. Papadimitriou, R. Sami, and S. Shenker, Distributed Computing 18 (2005), pp. 61-72. (Special issue of selected papers from Proc. of ACM PODC'02.)

Do incentives build robustness in BitTorrent? M.  Piatek, T. Isdal,
T.Anderson, A. Krishnamurthy, A. Venkataramani. NSDI 2007 

  Barry Burton


  Byzantine Techniques

The Byzantine Generals Problem.  Lamport et al.  ACM TOPLAS 4, 1982.

Randomized Byzantine Generals. Rabin. FOCS, 1983.

Byzantine Quorum Systems. Malkhi and Reiter.

Fault-scalable Byzantine Fault-Tolerant Services. Michael Abd-El-Malek et.al.  SOSP 2005

Practical Byzantine Fault Tolerance.  Castro and Liskov. 3rd OSDI, Feb 1999.



  How to prepare a cs614 presentation

  Each time we meet, we’ll start with a fairly short presentation, lasting perhaps 45 minutes, after which there will be a discussion lasting an additional half hour or so.  Preparing the lecture is a significant challenge and you should expect to spend as much as a week or two working on it before the date of your presentation. 

  For each date, there are several suggested references.  Any one of these papers could be the subject of a full-length lecture in itself, and several would take multiple lectures to cover in any depth.  Accordingly, you face a tough choice: either learn to speak at an extremely rapid rate, which will be completely incomprehensible, or select within the material, focusing on a core aspect or idea and leaving the surrounding material for others to either absorb simply by virtue of having done the readings, or to raise during the discussion. 

  Your choice of material and approach to the presentation should satisfy several properties:

  1)      Within 45 minutes you only have time to convey a very small number of technical points.  For example, you probably have time to state a single theorem and to present its proof, but even if the paper contains many theorems, unless they are very simple you won’t have time to talk about more than one (perhaps two).

  2)      Every talk needs to be placed in context.  You need to plan on starting by spending at least a few minutes (10 would be a good estimate) explaining what the problem attacked by the authors was, why this was an important problem (what alternative ideas were being advanced at the time?) and what prior work people need to be aware of.  You should say enough about this prior work to place the results you’ll focus on into a broader setting.

  3)      You should tell us what you’ll be telling us, but in a non-technical way.  For example, if you are presenting the “consistent cuts” paper, give some intuition into what a consistent cut is all about before trying to define the idea formally.  As you move into the formal material, remind us of this intuition and show us how the formal definition captures the intuitive one.  Your job is to make the formalism seem natural.  If you are talking about a systems result, try and talk about the thing the authors are doing from a high level – why did this excite them, and the community?  What was the key question they were interested by, and what was their “angle”?

  4)      Now, perhaps 10 to 15 minutes into your “slot” you’ll get to the more detailed material.  This is where you might briefly explain a formalism and present a theorem and its proof, or lay out the major architectural pieces of a system and explain how they were put together.

  5)      In the second half of the technical part of your talk, you might illustrate the way that a proof works using a small example, or present the experimental data associated with a systems paper.  Systems people tend to be nearly obsessive about experimental data.  It should be presented lovingly with attention to the insights that can only be gained by really building something and putting it through its paces.  Where are the fundamental costs in an approach?  How well does it scale?  Are their little warts and hidden problems that the data reveals?

  6)      In the final part of your talk, you should wrap things together by reminding us what the original premise of the paper you presented was, how the authors attacked the problem, and what the data revealed about the limitations and successes of their approach.  Express your opinions at this stage.  For a theory talk, try and put the results back into the broader context.  Do they say something broad and powerful, or is the main insight narrow?

  Your talk will need to be on Powerpoint slides (or some similar system, if you prefer something else).  Powerpoint makes suggestions about font sizes but in fact you can certainly pack more on a slide than is possible using the intial choices.  Keep in mind that your audience has to be able to read the slides when projected on a screen at a sensible size!  Either make plastic slides, or have your material on a laptop that can connect to the overhead display system. 


  People who have never given formal talks before often make one of two kinds of mistakes: they estimate their own pace incorrectly, or they simply make far too many slides.  Generally, figure that you can cover one slide with text on it per minute.  About half of your material should be graphics or illustrations of some sort – people get bored with endless text.

  So, for a 45 minute talk, you would probably make up something like 30 to 35 slides of actual text and an addition 10 to 15 that are mostly pictures you’ll explain during the talk.  Often you can copy some of this material right out of the online versions of papers or from the web site of the author.  However, we probably won’t have a network connection in the lecture room, so your slides do need to be self contained.

  A fast speaker who is comfortable in front of a group and familiar with her slides could perhaps use as many as 60 slides for a 45 minute talk.  But she would need to have relatively little text on each slide to get away with this. 

  Feel free to stray from the papers I’ve suggested.  But do read the papers I suggested.  It may be wise to start reading them two weeks early, since many are tough, high-content papers with a lot of meat to them.  People attending the class really should read them too, but probably will do so the day before your talk and won’t have time to really pick away at the material and think hard about it.  So your job is to take them to a deeper level of understanding than they will have gotten in reading the papers quickly over a few beers.  And their job is to have read the papers carefully enough to contribute to a spirited discussion in class.