Experience with Grapevine: The Growth of a Distributed System

Paper by: Michael D. Schroeder, Andrew D. Birrel, and Roger M. Needham. Summary by: Jason D. Hartline.

Significance and Purpose

The purpose of this paper was to document the the Grapevine system with the giving perspective to its original design considerations and usage assumptions. It re-evaluated these assumptions and made suggestions for changes to accommodate common usage patterns.

Structure

Grapevine is a distributed replicated system for message distribution and resource location. It uses a set of dedicated servers. On each it runs a message server and a registration server.

Registration Servers

Registration servers contain complete registries. Either all the entries of a registry are on a given server or none of them are. Entries are group entries or individual entries.

Message Servers

Message servers receive messages for delivery to recipients and distribution lists. It buffers messages for recipients until that person requests the message. Any message server can accept any message for delivery.

Scaling

The Plan

The plan when grapevine was being built was that when more server power was required to service an expanding user base, more equally powerful servers could be added instead of making the existing servers more powerful.

The only portion of the system that would not scale was the configuration information that each server needed to be able to communicate with the other servers.

More users would be dealt with by adding more registries not making larger registries.

Problems

They did not foresee that some general interest distribution lists would scale proportionally to the number of total users. In this respect, the number of messages on a specific server increased and the frequencies of user add and removes on that server increased. To combat this, they suggest adding a layer of indirection that points to registries on other servers. This way, as more users join the distribution list, more registries can be added splitting up the work. As well, user adds and removals would be easier on small portions of the list.

Configuration Issues

They answered several questions on how to configure the registries and replicated servers. First, where are optimal places in the network to primary and secondary replication servers so as to maximize inbox message sharing, assure that the secondary servers all are equally loaded if a given primary server goes down. It would not work to have all items stored in a primary server duplicated on the same secondary server. However, this requirement directly conflicts with the need to put inboxes on the same servers for optimal message sharing. It is also important to put secondary replicas on the other size of unreliable links from the primary server. With this in mind, the paper suggests that the primary inbox on the message server itself, and a secondary inbox nearby with a third replication across the network.

Transparency

The goal of Grapevine was to seem to its users as a single reliable server. Users did; however, notice inconsistencies in replicated registries right after changes were made to registry contents. As well, it was difficult to provide the administrators with some useful tools because of the distributive nature of Grapevine. Some users also experienced Duplicated messages due to some servers being unreachable at the time the message was sent.

Mistakes

They mentioned some choices they made that were mistakes. These include:
  1. Auto re-mailing of inboxes when they are moved from one server to another.
  2. Assumed users do not want to know about system state.
  3. Group nesting made authentication slow.
  4. Grapevine has no knowledge of the network state or topology. It could not make decisions based on bandwidth, congestion, or network reliability.

Modifications

One key point of this paper was the suggested modifications. These are changes that they felt would make Grapevine perform better after seeing it in action.
  1. Send difference registry update messages for adding and removing users from distribution lists. This saves time in processing the add/remove request. This can be done for user distribution lists because the order of additions and removals does not matter. This method can not be used for all registry items because for some items, the order of operations is important.
  2. Cache authentication and access control queries.
  3. Use syntax of group names with the uparrow to be able to tell which items in an access group need to be further expanded.

Reliability

The Grapevine system was built around replication for reliability. They mentioned that situations could arise where a machine would disallow more connections because its storage space was mostly filled. These situations could cause deadlock; however, deadlock was never reported to have happened except when they deliberately created a deadlock situation. Except for the reported delays and possible inconsistencies, Grapevine seems to be very reliable.

Conclusions

This paper made a clear presentation of the implemented Grapevine system and gave good hindsight observations from their experience with Grapevine as to what could have been done differently.

Questions

  1. Why do they not think that inconsistencies are a big problem?