6 Layer Anatomy: what are the pieces of a layer?
This is a description of the standard pieces of a Ensemble layer. This
description is meant to serve as a general introduction the standard
``idioms'' that appear in layers. Because all layers follow the same
general structure, we present a single documentation of that structure, so
that comments in a layer describe what is particular to that layer rather
than repeating the features each has in common with all the others.
Comments on additional information that would be useful here would be
appreciated.
6.1 Design Goals
A design goal of the protocol layers is to include as little
Ensemble-specific infrastructure is present in the layers. For instance,
none of the layers embody notions of synchronization, messages operations,
of event scheduling. In fact, the only Ensemble-specific modules used by
layers are the Event and the View modules.
6.2 Notes
Some general notes on layers:
-
All layers are in single files.
-
Usually the only objects exported by a layer are the type ``header'' and
the value ``l''. When referred to outside of a layer, these values are
prefixed with the name of the layer followed by ``.'' and the name of
the object (either ``header'' or ``l''). For instance the STABLE layer is
referenced by the name ``Stable.l''.
6.3 Values and Types
Listed below are the values and types commonly found in a layer, listed in
the usual order of occurrence. For each object we give a description of
its typical use and whether or not it is exported out of the layer.
-
name : Local variable containing the name of the layer.
-
failwith : Typically this standard Objective Caml function is redefined to
prefix the failure message with the name of the layer. Sometimes it is
also redefined to dump the state of a layer.
-
header : Exported type of the header a layer puts on messages.
Layers that do not put headers on messages do not have this type defined.
The type is exported abstractly so it is opaque outside the layer. This
type is usually a disjoint union (variant record) of several different
types. Some example variants are:
-
NoHdr : Almost always one of the values is defined to be
NoHdr which is used for messages for which the layer does not put
on a (non-trivial) header
- Data : Put on application data messages from the layer above
- Gossip : Put on gossip messages (such as by a stability layer)
- Ack/Nak : Acknowledgement or negative acknowledgements
- Retrans : Retransmissions of application data messages
- View : List of members in a new view.
- Fail : List of members being failed.
- Suspect : List of members to be suspected.
- Block : Prepare to synchronize this group.
-
nohdr : exported variable. This is a variable that only occurs in a few
layers. It is always defined to be the value NoHdr of the header type
of the layer. It is exported so that the rest of the system can do
optimizations when layers put trivial headers on a message.
-
normCastHdr : exported variable. This is a variable that is used only
by special layers that may need to generate a valid header for this layer.
This variable should have no relevence to the execution of a layer.
-
state : Local type that contains the state of an instance of a
layer. Some layers do not yet use this, but eventually all of them will.
[layers that do not use a state varable have the state split up
amongst several local state variables]. The state then is referenced
through the local variable s. A field in a state record is
referred to by the Objective Caml syntax, s.field, where field is
the name of a field in the state record.
Some example field names used in layer states:
-
time : time of last (up) ETimer event seen
- next_sweep/next_gossip :
next time that I want to do something (such as retransmit
messages, synchronize clocks, ...)
- sweep : time between sweeping
- buf : buffer of some sort
- blocking/blocked : boolean for whether group is currently blocking
- ltime : the current ``logical time stamp'' (usually taken from
the view_id)
- max_ltime : the largest logical time stamp seen by this member
- seqno : some sequence number
- failed :
information on which members have failed (either a
list of ranks or a boolean vector indexed by rank)
- elected : do I think I am the coordinator?
General notes on fields:
-
fields with type vect: usually array with one entry for each member, indexed by rank
- fields with type map: usually mapping of members eid to some state on the member
- fields with ``up'' (or ``dn'') in the name: refers to some info kept on
events that are going up (or down)
- fields with ``cast'' (or ``send'') in the name: refers to state
kept about broadcasts (or sends)
- fields with ``buf'' in the name: refers to some buffer
- fields with ``dbg'' in the name: fields used only for debugging puposes
- fields with ``acct'' in the name: fields used only for keeping
track of tracing or accounting data for the layer
-
dump : Local function that takes a value of type state and
prints some representation for debugging purposes
-
member : Local type that only occurs in some layers. Defines
state kept for each member in a group. Typically, the layer's state will
have an array of member objects indexed by rank (and this field is usually
called members). The notes above fields in state records generally
apply to the fields in member types as well.
-
init : Initialization function for this layer. Takes two
arguments. The first is a tuple containing arguments specific to this
layer. The second is a view state record containing arguments general to
the protocol stack this layer is in. This function does any initialization
required and returns a state record for a layer instance.
Some example names of initialization parameters for layers:
-
sweep : float value of how often to carry out some action
(such as retransmitting messages or pinging other members)
- timeout : amount of time to use for some timeout (such as how
long to wait before kicking a non-responsive member out of a group)
- window : size of window to use for some buffer
-
hdlrs : Function initializing handlers for this layer. Takes two
arguments. The first is a state record for this layer. The second
is a record containing all the handlers the layer is to use for passing
events out of the layer. Return value is a set of handlers to be used for
passing events into this layer.