A message is called stable when all members have seen that message. Each member can decide what it means to ``have seen'' a message. This may be immediately after message delivery, after the message has been written to disk, or after some external action. At that point the application invokes the UGI ack downcall. The UGI downcall is an internal acknowledgement of the member to the protocol layer, and does not necessarily result in an immediate message back to the sender of the message.
Stability information is useful to several other layers and applications. The membership layer uses stability information for garbage collection of messages. The current flow control layer uses it to control the flow of messages through Horus. The causal delivery layer uses it occasionally when communicating across different groups. Some fault-tolerant applications delay external action after receiving a message until that message is stable (named safe delivery in the Transis system).
To track message stability, the STABLE layer periodically broadcasts a
report that includes the sequence numbers of the last messages that have
been acknowledged.
It also piggybacks some information on existing traffic.
The layer collects the stability information in what is called the stability matrix.
When multicasts are sent, some of our protocols
need to know about
-stability (when
members have seen
the message, where
may be less than the full set of group members).
In support of this, the stability matrix indicates not only full, but also
partial stability.
The stability matrix is local and
its
entry stores the number of messages sent by
which have been
acknowledged by
.
The stability matrix is shared between the users and the stability layer
itself, and updates to the stability matrix are reported using STABLE event
upcalls.